Acquisition  Research  Journal 

ation  of  the  Defense  Acquisition  University 


Consciousness,  Professionalishn, 


and  Technical  Excellence 

2016 

Edward  Hirsch  Research  Paper  Competition 

Presented  on  behalf  of  DAU  by: 


Defense  Acquisition  University 


April  2016  Vol.  23  No.  2  ISSUE 


The  High  Flying  Leadership 
Qualities:  What  Matters  the  Most? 

Col  Robert  L.  Tremaine,  USAF  (Ret.) 


Metrics-based  Risk  Assessment  and 
Management  of  Digital  Forensics 

Mehmet  Sahinoglu, 

MSgt  Stephen  Stockton,  USAF  (Ret.), 
Capt  Robert  M.  Barclay,  USAF  (Ret.), 
and  Scott  Morton 


Catalysts  of  Military  Innovation:  A  Case  Study 
of  Defense  Biometrics 

COL  Glenn  Voelz,  USA 


#eVAL  Uate:  Monetizing  Service  Acquisition 
Trade-offs  Using  the  Quality-Infused 
Price ®  Methodology 

Capt  Daniel  J.  Finkenstadt,  USAF,  and 
Lt  Col  Timothy  G.  Hawkins,  USAF  (Ret.) 


The  Defense  Acquisition  Professional  Reading  List 

The  American  Warfare  State:  The  Domestic 
Politics  of  Military  Spending 

Written  by  Rebecca  Thorpe 
Reviewed  by  Trevor  Taylor 


Article  List 
AR  J  Extra 


Defense  Acquisition  Research  Journal 

A  Publication  of  the  Defense  Acquisition  University 


Defense  Acquisition  University 


Mr.  Frank  Kendall 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 

Mr.  James  P.  Woolsey 

President,  Defense  Acquisition  University 

Research  Advisory  Board 


Mr.  Patrick  Fitzgerald 

Defense  Contract  Audit  Agency 


Mr.  Richard  Altieri 

Dwight  D.  Eisenhower  School  for  National 
Security  and  Resource  Strategy 

Dr.  Michelle  Bailey 

Defense  Acquisition  University 

Dr.  Don  Birchler 

Center  for  Naval  Analyses  Corporation 

Mr.  Kevin  Buck 

The  MITRE  Corporation 

Mr.  John  Cannaday 

Defense  Acquisition  University 

Dr.  John  M.  Colombi 

Air  Force  Institute  of  Technology 

Dr.  Neal  Couture 

The  George  Washington  University 

Dr.  Richard  Donnelly 

The  George  Washington  University 

Dr.  William  T.  Eliason 

Dwight  D.  Eisenhower  School  for  National 
Security  and  Resource  Strategy 


Dr.  Mary  C.  Redshaw 

Dwight  D.  Eisenhower  School  for  National 
Security  and  Resource  Strategy 

Editorial  Board 

Dr.  Larrie  D.  Ferreiro 

Chairman  and  Executive  Editor 

Dr.  J.  Ronald  Fox 

Harvard  Business  School 

Mr.  David  Gallop 

Defense  Acquisition  University 

Dr.  Jacques  Gansler 

University  of  Maryland 

RADM  James  Greene,  USN  (Ret.) 

Naval  Postgraduate  School 

Dr.  Mike  Kotzian 

Defense  Acquisition  University 

Dr.  Craig  Lush 

Defense  Acquisition  University 

Dr.  Andre  Murphy 

Defense  Acquisition  University 

Dr.  Christopher  G.  Pernin 

RAND  Corporation 


Dr.  William  A.  LaPlante 

The  MITRE  Corporation 


Dr.  Richard  Shipe 

Dwight  D.  Eisenhower  School  for  National 
Security  and  Resource  Strategy 

Dr.  Keith  Snider 

Naval  Postgraduate  School 

Dr.  John  Snoderly 

Defense  Acquisition  University 

Ms.  Dana  Stewart 

Defense  Acquisition  University 

Dr.  David  M.  Tate 

Institute  for  Defense  Analyses 

Dr.  Trevor  Taylor 

Cranfield  University  (UK) 

Mr.  Jerry  Vandewiele 

Defense  Acquisition  University 

Dr.  Roy  L.  Wood 

Defense  Acquisition  University 


ISSN  2156-8391  (print)  ISSN  2156-8405  (online) 

The  Defense  Acquisition  Research  Journal,  formerly  the  Defense  Acquisition  Review  Journal,  is  published  quarterly  by  the  Defense 
Acquisition  University  (DAU)  Press  and  is  an  official  publication  of  the  Department  of  Defense.  Postage  is  paid  at  the  U.S.  Postal  facility, 
Fort  Belvoir,  VA,  and  at  additional  U.S.  Postal  facilities.  Postmaster,  send  address  changes  to:  Editor,  Defense  Acquisition  Research 
Journal,  DAU  Press,  9820  Belvoir  Road,  Suite  3,  Fort  Belvoir,  VA  22060-5565.  Some  photos  appearing  in  this  publication  may  be  digitally 
enhanced. 

Articles  represent  the  views  of  the  authors  and  do  not  necessarily  reflect  the  opinion  of  DAU  or  the  Department  of  Defense. 


Defense  Acquisition  Research  Journal 

A  Publication  of  the  Defense  Acquisition  University 


Director,  Visual  Arts  &  Press 

Managing  E  ditor 
Deputy  Director, 
Visual  Arts  &  Press 

Assistant  Editor 

Production  Manager, 
Visual  Arts  &  Press 

Lead  Graphic  Designer 

Graphic  Designer,  Digital 
Publications 

Technical  Editor 

Associate  Editor 

Copy  Editor/Circulation 
Manager 

Multimedia  Assistant 

Editing,  Design,  and  Layout 


Randy  Weekes 

Norene  L.  Taylor 

Aleisha  R.  Jenkins-Bey 

Frances  Battle 

Diane  Fleischer 

Angie  Brownie 

Collie  J.  Johnson 

Michael  Shoemaker 

Debbie  Gonzalez 

Noelia  Gamboa 

The  C3  Group  & 

Schatz  Publishing  Group 


A  Publication  of  the  Defense  Acquisition  University 


April  2016  Vol.  23  No.  2  ISSUE  77 


CONTENTS  I  Featured  Research 


,122 

The  High  Flying  Leadership 
Qualities:  What  Matters  the 
Most? 

Co/  Robert  L.  Tremaine,  USAF  (Ret.) 


Like  many  U.S.  companies,  the  Department  of 
Defense  (DoD)  invests  in  leadership  development. 
The  DoD  recognizes  equal  benefits  and  has  insti¬ 
tuted  various  programs  to  enable  it.  Not  every  DoD 
organization  does  so  to  the  same  degree,  however. 


,152 

Metrics-based  Risk  Assessment 
and  Management  of  Digital 
Forensics 

Mehmet  Sahinoglu,  MSgt  Stephen  Stockton, 
USAF  (Ret.),  Capt  Robert  M.  Barclay,  USAF 
(Ret.),  and  Scott  Morton 


Digital  forensics  risk  (digital  evidence  assessment 
subject  to  given  standards)  is  managed  and 
quantified  using  a  risk  meter  algorithm  for 
calculating  risk  indices. 


Featured  Research 


,178 

Catalysts  of  Military  Innovation: 

A  Case  Study  of  Defense 
Biometrics 

COL  Glenn  Voelz ,  USA 

This  article  examines  the  catalysts  of  military  tech¬ 
nology  innovation,  using  a  case  study  of  defense 
biometrics. 


,202 

#eVALUate:  Monetizing  Service 
Acquisition  Trade-offs  Using 
the  Quality-Infused  Price® 
Methodology 

Capt  Daniel  J.  Finkenstadt,  USAF,  and 
Lt  Col  Timothy  G.  Hawkins,  USAF  (Ret.) 


This  research  proposes  a  method  to  leverage  the 
use  of  subjective  service  quality  in  both  selecting 
contractors  and  managing  their  performance. 


A  Publication  of  the  Defense  Acquisition  University 


April  2016  Vol.  23  No.  2  ISSUE  77 


CONTENTS  I  Featured  Research 


p  vii 

DAU  Alumni  Association 


P  viii 

From  the  Chairman  and  Executive  Editor 


PX 

Research  Agenda  2016-2017 


Professional  Reading  List 


The  American  Warfare  State:  The  Domestic  Politics  of  Military  Spending 
Written  by  Rebecca  Thorpe  and  reviewed  by  Trevor  Taylor 


Defense  ARJ  Guidelines  for  Contributors 


The  Defense  Acquisition  Research  Journal  (ARJ)  is  a  scholarly  peer-reviewed  journal  published  by  the 
Defense  Acquisition  University.  All  submissions  receive  a  blind  review  to  ensure  impartial  evaluation. 

p  241 

Call  for  Authors 


We  are  currently  soliciting  articles  and  subject  matter  experts  for  the  2016-2017  Defense  ARJ  print 
years. 

p  244 

Defense  Acquisition  University  Web  Site 


Your  online  access  to  acquisition  research,  consulting,  information,  and  course  offerings. 


DAU  ALUMNI  ASSOCIATION 

Join  the  Success  Network! 

The  DAU  Alumni  Association  opens  the  door  to  a  worldwide  network  of  Defense 
Acquisition  University  graduates,  faculty,  staff  members,  and  defense  industry 
representatives— all  ready  to  share  their  expertise  with  you  and  benefit  from  yours. 

Be  part  of  a  two-way  exchange  of  information  with  other  acquisition  professionals. 

•  Stay  connected  to  DAU  and  link  to  other  professional  organizations. 

•  Keep  up  to  date  on  evolving  defense  acquisition  policies  and  developments 
through  DAUAA  newsletters  and  symposium  papers. 

•  Attend  the  DAUAA  Annual  Acquisition  Community  Conference/Symposium 
and  earn  Continuous  Learning  Points  toward  DoD  continuing  education 
requirements. 

Membership  is  open  to  all  DAU  graduates,  faculty,  staff,  and  defense  industry 
members.  It’s  easy  to  join  right  from  the  DAUAA  Web  Site  at  www.dauaa.org. 

For  more  information  call  703-960-6802  or  800-755-8805, 
or  e-mail  dauaa2@aol.com. 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


FROM  THE 
CHAIRMAN  AND 
EXECUTIVE 
EDITOR 

Dr.  Larrie  D.  Ferreiro 


The  theme  for  this  edition  of  Defense 
Acquisition  Research  Journal  is 
“Strengthening  Cost  Consciousness, 
Professionalism,  and  Technical 
Excellence,”  which  is  the  iron  triangle  of  a 
robust  defense  acquisition  system.  It  is  also 
the  theme  of  this  year’s  annual  research 
paper  competition  sponsored  by  our  part¬ 
ner  organization,  the  Defense  Acquisition 
University  Alumni  Association. 

The  first-place  winner  is  Col  Robert  L. 
Tremaine’s,  USAF  (Ret.)  “The  High  Flying  Leadership  Qualities: 
What  Matters  the  Most?”  In  it,  he  evaluates  the  factors  of  leader¬ 
ship,  such  as  leading  by  example  and  communicating  effectively, 
which  mark  effective  organizations  and  form  the  basis  for  thought¬ 
ful  and  effective  development  programs  that  combine  training, 
practice,  and  cultivation. 

The  second-place  winner  is  “Metrics-based  Risk  Assessment  and 
Management  of  Digital  Forensics,”  by  Mehmet  Sahinoglu;  MSgt 
Stephen  Stockton,  USAF  (Ret.);  Capt  Robert  M.  Barclay,  USAF 
(Ret.);  and  Scott  Morton.  The  authors  contend  that  digital  forensics, 
a  relatively  new  field  for  assessing  evidence  for  a  variety  of  purposes, 
can  be  quantified  by  using  a  novel  method  that  calculates  the  risk 
index  for  the  digital  forensics  process. 
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The  other  two  papers  in  this  edition  are  “Catalysts  of  Military 
Innovation:  A  Case  Study  of  Defense  Biometrics”  by  COL  Glenn 
Voelz,  USA,  which  examines  doctrinal  innovation  and  warfighting 
strategies  as  catalysts  of  military  technology  innovation  using  a 
case  study  of  defense  biometrics;  and  “#eVALUate:  Monetizing 
Service  Acquisition  Trade-offs  Using  the  Quality-Infused  Price® 
Methodology”  by  Capt  Daniel  J.  Finkenstadt,  USAF,  and  Lt  Col 
Timothy  G.  Hawkins,  USAF  (Ret.),  which  proposes  a  method  to 
leverage  the  use  of  subjective  service  quality  in  both  selecting  con¬ 
tractors  and  managing  their  performance. 

The  featured  book  in  this  issue’s  Defense  Acquisition  Professional 
Reading  List  is  Rebecca  Thorpe’s  The  American  Warfare  State: 
The  Domestic  Politics  of  Military  Spending,  reviewed  by  Professor 
Trevor  Taylor  of  Cranfield  University  at  the  Defence  Academy  of 
the  United  Kingdom. 
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DAU  CENTER 
FOR  DEFENSE 
ACQUISITION 

RESEARCH  AGENDA  2016-2017 


This  Research  Agenda  is  intended  to  make  researchers  aware  of 
the  topics  that  are,  or  should  be,  of  particular  concern  to  the  broader 
defense  acquisition  community  within  the  federal  government, 
academia,  and  defense  industrial  sectors.  The  center  compiles  the 
agenda  annually,  using  inputs  from  subject  matter  experts  across 
those  sectors.  Topics  are  periodically  vetted  and  updated  by  the 
DAU  Center’s  Research  Advisory  Board  to  ensure  they  address 
current  areas  of  strategic  interest. 

The  purpose  of  conducting  research  in  these  areas  is  to  provide 
solid,  empirically  based  findings  to  create  a  broad  body  of  knowl¬ 
edge  that  can  inform  the  development  of  policies,  procedures,  and 
processes  in  defense  acquisition,  and  to  help  shape  the  thought  lead¬ 
ership  for  the  acquisition  community.  Most  of  these  research  topics 
were  selected  to  support  the  DoD’s  Better  Buying  Power  Initiative 
(see  http://bbp.dau.mil).  Some  questions  may  cross  topics  and  thus 
appear  in  multiple  research  areas. 

Potential  researchers  are  encouraged  to  contact  the  DAU  Director 
of  Research  (research@dau.mil)  to  suggest  additional  research 
questions  and  topics.  They  are  also  encouraged  to  contact  the 
listed  Points  of  Contact  (POC),  who  maybe  able  to  provide  general 
guidance  as  to  current  areas  of  interest,  potential  sources  of  infor¬ 
mation,  etc. 
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Competition  POCs 

•  John  Cannaday,  DAU:  john.cannaday@dau.mil 

•  Salvatore  Cianci,  DAU:  salvatore.cianci@dau.mil 

•  Frank  Kenlon  (global  market  outreach),  DAU:  frank. 
kenlon@dau.mil 

Measuring  the  Effects  of  Competition 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  effect  on  defense  acquisition  costs  of  maintaining 
the  defense  industrial  base  in  various  sectors? 

•  What  means  are  there  (or  can  be  developed)  of  mea¬ 
suring  the  effect  of  utilizing  defense  industrial 
infrastructure  for  commercial  manufacture,  and  in 
particular,  in  growth  industries?  In  other  words,  can 
we  measure  the  effect  of  using  defense  manufacturing 
to  expand  the  buyer  base? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  of  openness  that  exists  in  competitive 
awards? 

•  What  are  the  different  effects  of  the  two  best  value 
source  selection  processes  (trade-off  vs.  lowest  price 
technically  acceptable)  on  program  cost,  schedule,  and 
performance? 

Strategic  Competition 

•  Is  there  evidence  that  competition  between  system 
portfolios  is  an  effective  means  of  controlling  price 
and  costs? 

•  Does  lack  of  competition  automatically  mean  higher 
prices?  For  example,  is  there  evidence  that  sole  source 
can  result  in  lower  overall  administrative  costs  at  both 
the  government  and  industry  levels,  to  the  effect  of 
lowering  total  costs? 

•  What  are  the  long-term  historical  trends  for  compe¬ 
tition  guidance  and  practice  in  defense  acquisition 
policies  and  practices? 
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•  To  what  extent  are  contracts  being  awarded  non- 
competitively  by  congressional  mandate  for  policy 
interest  reasons?  What  is  the  effect  on  contract  price 
and  performance? 

•  What  means  are  there  (or  can  be  developed)  to  deter¬ 
mine  the  degree  to  which  competitive  program  costs 
are  negatively  affected  by  laws  and  regulations  such  as 
the  Berry  Amendment,  Buy  America  Act,  etc.? 

•  The  DoD  should  have  enormous  buying  power  and  the 
ability  to  influence  supplier  prices.  Is  this  the  case? 
Examine  the  potential  change  in  cost  performance 
due  to  greater  centralization  of  buying  organizations 
or  strategies. 

Effects  of  Industrial  Base 

•  What  are  the  effects  on  program  cost,  schedule,  and 
performance  of  having  more  or  fewer  competitors? 
What  measures  are  there  to  determine  these  effects? 

•  What  means  are  there  (or  can  be  developed)  to  measure 
the  breadth  and  depth  of  the  industrial  base  in  various 
sectors  that  go  beyond  simple  head-count  of  providers? 

•  Has  change  in  the  defense  industrial  base  resulted  in 
actual  change  in  output?  How  is  that  measured? 

Competitive  Contracting 

•  Commercial  industry  often  cultivates  long-term,  exclu¬ 
sive  (noncompetitive)  supply  chain  relationships.  Does 
this  model  have  any  application  to  defense  acquisition? 
Under  what  conditions/circumstances? 

•  What  is  the  effect  on  program  cost,  schedule,  and 
performance  of  awards  based  on  varying  levels  of 
competition:  (a)  “Effective”  competition  (two  or  more 
offers);  (b)  “Ineffective”  competition  (only  one  offer 
received  in  response  to  competitive  solicitation);  (c) 
split  awards  vs.  winner  take  all;  and  (d)  sole  source. 
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Improve  DoD  Outreach  for  Technology  and  Products 
from  Global  Markets 

•  How  have  militaries  in  the  past  benefitted  from  global 
technology  development? 

•  How/why  have  militaries  missed  the  largest  techno¬ 
logical  advances? 

•  What  are  the  key  areas  that  require  the  DoD’s  focus  and 
attention  in  the  coming  years  to  maintain  or  enhance 
the  technological  advantage  of  its  weapon  systems  and 
equipment? 

•  What  types  of  efforts  should  the  DoD  consider  pursu¬ 
ing  to  increase  the  breadth  and  depth  of  technology 
push  efforts  in  DoD  acquisition  programs? 

•  How  effectively  are  the  DoD’s  global  science  and  tech¬ 
nology  investments  transitioned  into  DoD  acquisition 
programs? 

•  Are  the  DoD’s  applied  research  and  development  (i.e., 
acquisition  program)  investments  effectively  pursuing 
and  using  sources  of  global  technology  to  affordably 
meet  current  and  future  DoD  acquisition  program 
requirements?  If  not,  what  steps  could  the  DoD  take 
to  improve  its  performance  in  these  two  areas? 

•  What  are  the  strengths  and  weaknesses  of  the  DoD’s 
global  defense  technology  investment  approach  as 
compared  to  the  approaches  used  by  other  nations? 

•  What  are  the  strengths  and  weaknesses  of  the  DoD’s 
global  defense  technology  investment  approach  as 
compared  to  the  approaches  used  by  the  private  sec¬ 
tor— both  domestic  and  foreign  entities  (companies, 
universities,  private -public  partnerships,  think  tanks, 
etc.)? 

•  How  does  the  DoD  currently  assess  the  relative  benefits 
and  risks  associated  with  global  versus  U.S.  sourcing 
of  key  technologies  used  in  DoD  acquisition  programs? 
How  could  the  DoD  improve  its  policies  and  procedures 
in  this  area  to  enhance  the  benefits  of  global  technology 
sourcing  while  minimizing  potential  risks? 
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•  How  could  current  DoD/U.S.  Technology  Security  and 
Foreign  Disclosure  (TSFD)  decision-making  policies 
and  processes  be  improved  to  help  the  DoD  better  bal¬ 
ance  the  benefits  and  risks  associated  with  potential 
global  sourcing  of  key  technologies  used  in  current  and 
future  DoD  acquisition  programs? 

•  How  do  DoD  primes  and  key  subcontractors  currently 
assess  the  relative  benefits  and  risks  associated  with 
global  versus  U.S.  sourcing  of  key  technologies  used  in 
DoD  acquisition  programs?  How  could  they  improve 
their  contractor  policies  and  procedures  in  this  area 
to  enhance  the  benefits  of  global  technology  sourcing 
while  minimizing  potential  risks? 

•  How  could  current  U.S.  Export  Control  System  deci¬ 
sion-making  policies  and  processes  be  improved  to 
help  the  DoD  better  balance  the  benefits  and  risks 
associated  with  potential  global  sourcing  of  key  tech¬ 
nologies  used  in  current  and  future  DoD  acquisition 
programs? 

Comparative  Studies 

•  Compare  the  industrial  policies  of  military  acquisition 
in  different  nations  and  the  policy  impacts  on  acquisi¬ 
tion  outcomes. 

•  Compare  the  cost  and  contract  performance  of  highly 
regulated  public  utilities  with  nonregulated  “natural 
monopolies,”  e.g.,  military  satellites,  warship  building, 
etc. 

•  Compare  contracting/competition  practices  between 
the  DoD  and  complex,  custom-built  commercial  prod¬ 
ucts  (e.g.,  offshore  oil  platforms). 

•  Compare  program  cost  performance  in  various  market 
sectors:  highly  competitive  (multiple  offerors),  limited 
(two  or  three  offerors),  monopoly? 

•  Compare  the  cost  and  contract  performance  of  mil¬ 
itary  acquisition  programs  in  nations  having  single 
“purple”  acquisition  organizations  with  those  having 
Service-level  acquisition  agencies. 
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ership  development.  The  DoD  recognizes  equal  benefits  and  has  instituted 
various  programs  to  enable  it.  However,  not  every  DoD  organization  invests 
in  leadership  development  the  same  way.  The  ones  that  do  think  more  deeply 
about  their  future  have  thoughtful  and  effective  leadership  development 
programs  that  combine  training,  practice,  and  cultivation— all  intended  to 
professionally  nurture  future  leaders.  DoD  organizations  that  have  more 
defined  hierarchical  leadership  structures  such  as  mid-level  managers  (MLMs), 
senior-level  managers  (SLMs),  senior-level  leaders  (SLLs),  or  equivalent  offer 
a  gateway  to  learn  more  about  what  leadership  qualities  matter  to  them.  At 
the  Defense  Acquisition  University,  37  MLMs,  and  32  SLMs  provided  valu¬ 
able  insights  in  their  survey  responses.  No  SLLs  participated  in  this  study. 
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Keywords:  leadership  qualities,  DoD,  DAU,  senior-  and  mid-level  managers,  strategic  planning 
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If  institutions  like  DAU  treat  learning  as  a  lifetime  pursuit,  then  what 
do  its  mid-level  managers  (MLMs)  and  senior-level  managers  (SLMs)  have 
to  say  about  the  leadership  qualities  that  matter  most  during  their  own 
continuing  professional  development  journey?  Aside  from  growing  more 
capable  leaders  along  with  the  ability  to  create  greater  influence  inside  and 
outside  their  learning  spheres,  are  there  any  leadership  quality  outliers  in 
particular  that  deserve  a  more  intensive  review  based  on  responses  from 
a  representative  sample  population?  The  DAU  workforce  is  in  a  powerful 
position  to  address  this  question  given  the  inherent  diversity  and  capa¬ 
bility  among  its  ranks,  as  well  as  the  previous  operational  and  functional 
background  of  its  personnel  steeped  in  both  DoD  and  industry  experience. 


Research  Methodology 

Based  on  their  experiences,  survey  respon¬ 
dents  were  asked  to  identify  the  five  leadership 
qualities  that  mattered  most  to  them,  from 
a  list  of  14  representative  ones  drawn  from 
multiple  sources.  The  respondents  had  to 
make  hard  choices.  What  specific  factors 
influenced  their  leadership  quotient  and 
why?  Did  their  position,  generational 
affiliation,  supervisory  experience, 
and  number  of  years  in  their  current 
position  at  DAU  create  any  notice¬ 
able  flux?  What  about  the  qualities 
that  fell  outside  their  top  five? 

Were  they  still  important,  and 
to  what  degree?  The  remain¬ 
der  of  this  article  addresses 
answers  to  these  questions  in 

aggregate,  as  well  as  in  the  context  of  various  demographic 
slices  among  both  MLMs  and  SLMs  to  understand  better  the 
causes,  and  whether  or  not  there  is  a  cause  for  concern  for  other  insti¬ 
tutions  similar  to  DAU  throughout  the  DoD.  The  more  granular  results  are 
reported  through  frequency  tables  and  augmented  by  qualitative  comments. 


The  order  of  the  14  leadership  qualities  (Table  1)  in  this  particular  survey 
was  intentionally  randomized. 
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1  TABLE  1. 14  LEADERSHIP  QUALITIES  1 

Leads  by  Example 

Develops  Self  &  Others 

Effective  Communicator 

Builds  &  Nurtures  Trust 

Relationships 

Competent 

Credible 

Displays  Respect  &  Support  for 
Others 

Behavior  Aligns  with  DAU  Values 

Critical  Thinking 

Exercises  Authority  &  Decision 

Making 

Promotes  Collaboration 

Maintains  DAU  Enterprise 

Perspective 

Change  Agent 

Innovator 

Results  and  Findings  (Aggregate) 

The  Figure  displays  Aggregate  Survey  Results.  Among  all  the  respon¬ 
dents,  Leads  by  Example  and  Effective  Communicator  rose  as  the  top 

two  choices.  Research  underscores  similar  findings.  Both  characteris¬ 
tics  seem  to  embody  the  importance  of  the  expected  qualities  found  “in” 
and  “of”  leaders;  they  also  tend  to  be  inextricably  linked  in  practice.  RBC 
Financial  Group,  Canada’s  largest  financial  corporation,  recognized  the 
value  and  combined  the  two  by  instituting  a  communication  process  called 
“Leadership  Dialogues”  where  “established  leaders  relate  their  career 
experiences  to  developing  leaders”  (Beslin  &  Reddin,  2004).  As  part  of 
Effective  Communicator,  listening  is  also  an  especially  important  compo¬ 
nent.  Listening  takes  time  and  generally  requires  us  to  think  more  about 
our  thinking  (i.e.,  metacognition).  Without  it,  decision  missteps  can  poten¬ 
tially  result.  In  their  book.  Leadership  by  Example:  The  Ten  Key  Principles 
of  All  Great  Leaders,  Dr.  Sanjiv  Chopra  and  David  Fisher  remind  us  that  as 
Abraham  Lincoln  said,  “It  is  better  to  be  silent  and  be  thought  a 
fool  than  to  speak  up  and  dispel  all  doubt”  (Chopra  &  Fisher, 
2012)  by  speaking  up  too  soon.  Surprisingly,  research  shows 
the  average  person  listens  at  around  25  percent  efficiency 
levels  (Huscman,  Lahiff,  &  Penrose,  1988),  even  though  lis¬ 
tening  is  so  closely  tied  to  effective  leadership  (Johnson  & 
Bechler,  1998).  An  ample  supply  of  programs  teaches  us  to  be 
better  communicators;  few  programs  exist  that  teach  us  to  be 
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better  listeners  (Janusik,  Fullenkamp,  &  Partese,  n.d.)  or  the  important  role 
that  culture  plays  in  communication  through  the  motivation,  knowledge, 
and  skills  of  the  interactants  involved  (Spitzberg,  1994). 


FIGURE.  AGGREGATE  SURVEY  RESULTS 


Leads  by  Example 

EFFECTIVE  COMMUNICATOR 

Develops  Self  &  Others 
Builds  &  Nurtures  Trust  Relationships 

CREDIBLE 

Competent 

DISPLAYS  RESPECT  &  SUPPORT  FOR  OTHERS 

Critical  Thinker 
BEHAVIOR  ALIGNS  WITH  DAU  VALUES 

Exercises  Authority  &  Decision  Making 
Promotes  Collaboration 
Maintains  DAU  Enterprise  Perspective 
Innovator 
Change  Agent 

0%  15%  30%  45%  60%  75%  90% 


For  Develops  Self  and  Others,  instruments  like  an  organization’s 
Strategic  Plan  (SP)  or  other  similar  means  generally  characterize  some 
aspect  of  its  leadership  development  programs  as  part  of  its  mission  heading. 
The  Department  of  Homeland  Security  (DHS,  2014)  addresses  leader¬ 
ship  development  in  its  SP  under  Goal  6:  Strengthen  Service  Delivery  and 
Manage  DHS  Resources,  with  a  specific  objective  that  focuses  on  “building 
an  effective,  mission-focused,  diverse,  and  inspiring  cadre  of  leaders”  (p.  45). 
Whatever  the  manifestation,  these  programs  can  also  pay  huge  dividends  by 
lowering  costly  turnover  rates,  growing  more  capable  leaders,  and  creating 
greater  opportunities  for  professional  gains  as  well  as  concomitant  organi¬ 
zational  successes  inside  and  outside  their  domains. 
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In  the  respondents’  selection  of  their  top  five,  the  author  discerned  a 
noticeable  variance  between  how  DAU  SLMs  and  MLMs  viewed  Effective 
Communication,  Credible,  and  Displays  Respect  for  Others.  MLMs 
more  often  selected  Effective  Communication  and  Displays  Respect  for 
Others  in  their  top  five,  and  provided  quite  a  few  supporting  comments  to 
reinforce  their  importance: 

Effective  Communication:  “A  leader  must  be  able  to  com¬ 
municate  vision/purpose  to  the  organization  for  it  to 
understand  goals  and  why  they  are  important  to  the  mission 
....  Basis  for  leadership ....  Can’t  lead  if  you  can’t  communicate 
....  It’s  not  WHAT  you  say,  but  HOW  you  say  it ....  Effective 
leaders  must  be  able  to  share  knowledge  and  ideas  as  well 
as  transmit  urgency  and  enthusiasm  to  others.” 

Displays  Respect  for  Others  (Critical  for  a  Leader):  “Treat 
others  the  way  you  would  expect  to  be  treated  ....  A  leader 
needs  to  respect  not  only  the  people  that  work  for  them,  but 
also  everyone  in  the  enterprise;  otherwise,  trust  breaks 
down  ....  A  successful  organization  demonstrates  respect 
for  all  levels  of  the  organization ....  Without  respect,  others 
will  not  listen  or  follow.  An  effective  leader  must  be  willing 
to  consider  others’  opinions  and  be  open  to  feedback,  even 
if  it’s  not  favorable.” 

SLMs  placed  a  greater  emphasis  than  MLMs  on  Credible.  For  some  MLMs, 
Credible  may  have  dropped  out  of  their  top  five  based  on  their  supporting 
comments  (found  under  Leads  by  Example  and  Competent)  where  they 
responded: 

Credible:  “Basis  of  credibility ...  A  subordinate  should  only 
have  to  look  one  place  for  the  standard  that  needs  to  be  met— 
the  supervisor ....  Do  as  I  do  works  much  better  than  do  as  I 
say ....  Time  honored  leadership  quality ...  It’s  one  of  the  key 
things  I  look  for  in  my  leaders  ....  You  get  from  others  what 
you  model  for  them ....  You  must  be  an  expert  in  your  chosen 
field— it  ties  to  credibility.” 

Involving  more  MLMs  as  “leads”  on  strategic  initiatives  that  cut  across 
the  enterprise,  where  they  can  demonstrate  how  their  dependability  and 
expertise  converge,  might  help  close  the  gap  between  the  SLMs  and  MLMs 
top  five. 
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Findings  (By  Demographic) 

Does  a  leader’s  role  (either  faculty  or  staff)  influence  the  importance  of 
certain  leadership  qualities? 

For  each  role  grouping,  the  following  leadership  qualities  rose  one  standard 
deviation  above  the  mean  (x  +  Ict)  as  shown  in  Table  2. 
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•  Associate  deans:  Develops  Self  and  Others,  Credible,  Critical 
Thinking,  and  Innovator 

•  SLM  faculty:  Leads  by  Example,  Competent,  Behavior 
Aligns  with  DAU  Values,  Promotes  Collaboration,  and 
Maintains  DAU  Enterprise  Perspective 

•  MLM  staff:  Effective  Communicator,  Builds  and  Nurtures 
Trust  Relationships,  Competent,  Maintains  DAU  Enterprise 
Perspective,  and  Innovator 

•  SLM  staff:  Leads  by  Example,  Promotes  Collaboration,  and 

Change  Agent 

For  the  same  grouping,  the  following  leadership  qualities  fell  one  standard 
deviation  below  the  mean  (x  -  la)  as  shown  in  Table  2. 

•  Associate  deans:  Competent  and  Promotes  Collaboration 

•  MLM  staff:  Leads  by  Example,  Credible,  Critical  Thinking, 
and  Change  Agent 

•  SLM  faculty:  Effective  Communicator,  Builds  and  Nurtures 
Trust  Relationships,  Credible,  Change  Agent,  and  Innovator 

•  SLM  staff:  Effective  Communicator  and  Innovator 

For  Credible,  the  foundation  of  building  trust,  according  to  Stephen  Covey 
(2009),  MLM  faculty  who  raised  its  importance  responded: 

“Can’t  lead  without  it ...  Captures  a  number  of  the  other 
qualities  that  matter  and  would  be  foolish  to  leave  it 
out ...  implies  knowledgeable  and  proactive  ...  Similar  to 
Competent— means  we  bring  experience  to  the  situation 
...  When  subordinates  come  to  believe  that  a  senior  is  not 
credible  or  sufficiently  informed,  not  honest,  forthright,  or 
responsive,  they’ll  likely  no  longer  be  listening  by  the  time 
the  leader  finally  recognizes  his/her  isolation.” 

F or  Change  Agent,  where  leaders  work  to  alter  employee  attitudes  and  behav¬ 
iors  because  it’s  important  for  long-term  success  and  sustainability  (Abbas  & 
Asghar,  2010,  p.  26),  SLM  staff  who  raised  its  importance  responded: 
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“Change  is  the  constant ....  Need  Change  Agent  to  overcome 
natural  resistance  to  change  ...  DAU  can’t  continue  to  do 
things  the  way  they  have  always  been  done ....  Our  leadership 
needs  to  be  able  to  recognize  positive  change  and  be  willing 
to  accomplish  that  change.” 


Do  generational  affiliations  indi¬ 
cate  any  predispositions? 

For  the  generational  slice,  the  fol¬ 
lowing  leadership  qualities  rose 
one  standard  deviation  above  the 
mean  (x  +  Ict)  as  shown  in  Table  3. 


Some  of  the  greatest  differences  in  the  top  five  selections  occurred  among 
the  associate  deans,  SLM  faculty,  and  MLM  staff  in  their  selections  of 
Competent,  Credible,  Promotes  Collaboration,  and  Innovator.  Of  all  the 
demographic  groups,  the  associate  deans  were  the  only  one  to  score  Develops 
Self  and  Others  one  standard  deviation  above  the  mean.  As  the  saying  goes, 
“What  you  see  depends  on  where  you  sit.”  Associate  deans  might  be  more 
strategically  positioned  to  witness 
the  greater  impacts  that  a  more 
capable  and  “developed”  workforce 
can  make.  SLM  faculty  were  the 
only  group  to  raise  Behavior  Aligns  with 
DAU  Values  one  standard  deviation  above  the 
mean.  This  might  stem  from  their  frequent  interac¬ 
tion  with  diversified  and  sometimes  larger  groups, 
combined  with  the  recognition  that  “the 
greater  the  linkage  between  behavior  and 
values,  the  greater  an  organization’s  suc¬ 
cess”  (Rubino,  1998).  SLM  faculty  also 
generally  witness  firsthand  the 
prevailing  professionalism, 
enthusiasm,  and  resulting 
impacts  of  more  cross-cut- 
ting  enterprise  projects  (or  the 
absence  thereof),  firsthand. 
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•  GEN  X  SLMs:  Competent,  Critical  Thinking,  Exercises 
Authority  and  Decision  Making,  and  Change  Agent 

•  GEN  X  MLMs:  Effective  Communicator  and  Builds  and 
Nurtures  Trust  Relationships 

In  this  same  category,  the  following  leadership  qualities  fell  one  standard 
deviation  below  the  mean  (x  -  la)  as  shown  in  Table  3. 

•  Boomer  SLMs:  Competent 

•  GEN  X  SLMs:  Develops  Self  and  Others,  Effective 
Communicator,  Builds  and  Nurtures  Trust  Relationships, 
Displays  Respect  for  Others,  Behavior  Aligns  with  DAU  Values, 
Maintains  DAU  Enterprise  Perspective,  and  Innovator 

•  GEN  X  MLMs:  Credible,  Exercises  Authority  and  Decision 
Making,  and  Change  Agent 

By  juxtaposing  SLMs  and  MLMs  along  the  lines  of  their  generational  affili¬ 
ation,  more  dramatic  variances  surfaced  for  GEN  X  in  particular.  While  the 
Boomers  were  generally  consistent  in  the  selection  of  their  top  five,  GEN  X 
SLMs’  selections  were  more  dispersed  for  11  of  the  14  qualities,  while  GEN  X 
MLMs  were  less  distributed  in  their  selections.  No  Boomer  left  any  of  the  14 
qualities  out  of  their  top  five  leadership  qualities.  GEN  X  SLMs  and  GEN  X 
MLMs  left  out  four  and  two,  respectively.  There  can  be  several  explanations 
for  the  GEN  X  fluctuations. 

GEN  X  SLMs  apparently  placed  significantly  more  stock  in  Competent, 
Critical  Thinking,  Exercises  Authority  and  Decision  Making,  and  Change 
Agent  in  what  appears  to  be  at  the  expense  of  three  of  the  top  five. 

For  Competent,  they  may  have  learned  and  want  what  Kolditz  (2007)  the¬ 
orized:  “Leaders  need  to  take  the  time  and  effort  to  show  followers  what 
they’re  good  at  and  why  followers  should  be  confident  in  the  leader’s  ability” 
(p.  41).  In  their  supporting  comments,  the  respondents  said: 

Competent:  “A  leader  needs  to  be  competent  for  several 
reasons.  Subordinates  will  have  respect  for  a  leader  that  has 
technical  and  leadership  competence ....  A  competent  leader 
automatically  sets  high  standards  for  his/her  employees 
because  subordinates  will  naturally  follow  leadership’s 
example ....  A  leader  should  be  competent  in  their  role;  if  not, 
then  that  is  a  weakness  to  those  you  wish  to  lead.” 
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For  Critical  Thinking,  they  could  have  learned  very  early  the  value  of  ques¬ 
tioning  more,  challenging  the  status  quo,  and  reaping  the  benefits  of  creative 
tension  and  divergent  thinking.  They  may  have  even  learned  how  to  “dis¬ 
pute  their  beliefs,”  according  to  Dr.  Albert  Ellis,  and  promote  more  rational 
thinking  about  their  own  beliefs  (Epstein,  2001);  as  well  as  recognize  what 
other  scholars  have  reported— that  thinking  controls  feelings  and  volition 
(Elder,  1996),  which  can  easily  cloud  rational  and  sound  thinking.  Two  of 
the  respondents  pointed  out  that: 

Critical  Thinking:  “Critical  thinking  skills  are  required 
for  an  individual  to  be  successful  at  nearly  all  of  the  quali¬ 
ties  identified ....  It  strengthens  individual  capabilities  and 
encourages  professionalism  of  others  through  an  intellec¬ 
tually  disciplined  process  by  conceptualizing,  applying, 
evaluating,  and  formulating  a  reasoning  of  beliefs.” 

For  Exercises  Authority  and  Decision  Making,  the  other  groups  gave  a  sub¬ 
stantially  higher  number  of  reasons  for  keeping  them  out  of  the  top  five.  Two 
respondents  characterized  it  simply  by  saying: 

Exercises  Authority  and  Decision  Making:  “I  feel  this  trait  is 
important,  but  not  as  valuable  as  others  listed  in  my  opinion 
....  I  considered  it  less  important  [and]  because  I  have  to  trust 
my  people  to  execute,  I  delegate.” 

For  Change  Agent  to  rise  in  the  ranking,  especially  in  the  top  five,  some¬ 
thing  had  to  occur  with  some  of  the  GEN  X  MLMs  in  their  past  where  they 
probably  experienced  the  necessity  for  change.  More  often  than  not,  many 
individuals  generally  question  the  need.  Why  the  change?  How  will  I/we 
be  affected?  Am  I/we  at  risk  as  a  result  of  the  change?  Harvard  Professor 
John  Hotter  (1996)  established  an  eight-step  process  if  the  case  for  change 
can  be  made. 

1.  Establishing  a  Sense  of  Urgency 

2.  Creating  the  Guiding  Coalition 

3.  Developing  a  Vision  and  Strategy 

4.  Communicating  the  Change  Vision 

5.  Empowering  Employees  for  Broad-based  Action 

6.  Generating  Short-term  Wins 
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7.  Consolidating  Gains  and  Producing  More 
Change 

8.  Anchoring  New  Approaches  in  the  Culture 

Kotter’s  construct  is  still  very  popular.  However,  GEN 
X  MLMs  might  not  yet  fully  appreciate  the  extent  of  the 
value  proposition  of  change  due  to  inexperience  and/or  lim¬ 
ited  exposure  to  certain  situations,  the  reason  for  change,  or 
perhaps  merely  more  inconsistency  among  the  MLMs  in 
their  top  five  selections.  Timing  could  also  be  a  factor. 

For  example,  DAU  underwent  a  major  transformation  at 
the  turn  of  the  past  century.  DAU’s  relevancy  as  an  insti¬ 
tution  came  under  scrutiny.  It  was  about  to  be  absorbed 
by  another  institution.  DAU  clearly  had  a  “Sense  of 
Urgency”  (i.e.,  Kotter’s  Step  1)  and  even  incorporated  the 
word  “Transformation”  to  promulgate  it  as  one  of  DAU’s  five 
top  goals.  DAU  had  to  change— and  many  of  the  Boomers  and  some 
GEN  X  SLMs  took  part  in  the  transformation.  GEN  X  MLMs  who 
joined  DAU  later  didn’t,  and  missed  the  revolution.  “Transformation” 
is  no  longer  a  DAU  Strategic  Goal,  which  could  later  create  greater 
resistance  to  change. 

The  reason  GEN  X  MLMs  ranked  Credible  so  low  is  only  speculative. 
Instead  of  devaluing  Credible,  they  may  have  made  tighter  connections  to 
other  leadership  qualities.  One  of  the  respondents  said,  “Credible  is  similar 
to  competent— it  means  we  bring  experience  to  the  situation.”  Additionally, 
GEN  X  MLMs  may  not  fully  appreciate  the  trust  tax  (Covey,  2009)— imposed 
by  certain  leaders  (and  organizations)  and  so  closely  coupled  with  credibil¬ 
ity— that  costs  organizations  time  and  money  by  instituting  (or  inadvertently 
maintaining)  various  decision  barriers  (e.g.,  lengthy  coordination  cycles, 
bureaucratic  red  tape,  extensive  time  spent  in  meetings,  etc.). 

Does  supervisory  experience  influence  the  perceived  importance  of  certain 
leadership  qualities? 


For  SLMs  with  15  years  or  more  of  supervisory  experience,  the  following 
leadership  qualities  rose  one  standard  deviation  above  the  mean  (x  +  Ict)  as 
shown  in  Table  4,  with  the  following  supporting  comments: 
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Develops  Self  and  Others:  “[A]  Leader’s  job  is  to  train  him/ 
herself  out  of  their  jobs  by  preparing  the  next  wave  of  lead¬ 
ers  and  prepare  themselves  for  their  next  job  ....  Learning 
is  a  never-ending  process.  Everyone  can  always  improve, 
learn  something  new,  and  expand  their  minds,  thoughts, 
and  ideas.  This  will  lead  to  better  critical  thinking  and 
open  up  peoples’  ‘apertures’  as  they  view  the  world  ....  One 
of  the  most  important  functions  of  a  leader  is  to  facilitate 
development  of  his  or  her  subordinates,  providing  mentor¬ 
ship  and  development  opportunities  so  they  can  accomplish 
success  in  their  own  careers  and  positively  contribute  to  the 
mission ....  We  have  to  stay  current  and  relevant,  and  we  have 
to  do  succession.  That  means  developing  our  people,  and 
also  giving  them  [the]  best  chance  to  succeed  also  outside 
of  DAU  ....  Enabling  opportunities  for  growth  in  capability 
and  improvement  in  themselves  by  supporting  learning 
engagements  and  new  experiences  demonstrates  direct 
interest  and  investment  in  the  individual  that  coincides 
with  objectives  of  the  organization.” 

Credible:  “A  credible  leader  possesses  character  (ethical, 
honest,  loyal,  respects  others)  and  is  recognized  as  com¬ 
petent  (accountable  and  gets  results)  ....  Credibility  is  the 
foundation  for  effectiveness  and  working  with  others  as 
senior,  peer,  or  subordinate;  credibility  includes  competency 
....  A  lack  of  respect  and  support  for  others  severely  degrades 
the  organizational  climate ....  Most  important  quality.  Goes 
with  integrity.  Without  it,  there  will  be  no  trust.” 

For  SLMs  with  less  than  15  years  of  supervisory  experience,  the  following 
leadership  qualities  rose  one  standard  deviation  above  the  mean  (x  +  lc), 
with  the  following  supporting  comments: 

Displays  Respect  and  Support  for  Others:  “This  is  a 
simple  rule,  but  often  forgotten.  Simple  respect  for  everyone, 
regardless  of  rank  or  position.  It  is  just  as  important  to  treat 
the  janitorial  staff  with  respect  as  it  is  senior  leadership— 
everyone  deserves  respect.  The  truth  is  we  all  just  have 
different  jobs.  This  rolls  into  leading  by  example— people 
watch  the  way  you  treat  others  and  it  makes  a  difference  on 
how  they  see  you  as  leadership  material.” 
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Exercises  Authority  and  Decision  Making:  None  given. 

Promotes  Collaboration:  “This  piggy-backs  on  DAU 
Values— we  are  customer-focused,  team-oriented,  strive  for 
excellence,  and  are  agile  and  responsive  to  customer  require¬ 
ments  ....  We  must  promote  collaboration  with  our  faculty 
peers,  stakeholders,  acquisition  workforce,  etc.,  to  ensure 
that  we  develop  the  most  qualified  acquisition  workforce.” 

Change  Agent:  None  given. 

For  MLMs  with  less  than  15  years  of  supervisory  experience,  the  following 
leadership  quality  rose  one  standard  deviation  above  the  mean  (x  +  Ict)  as 
shown  in  Table  4,  with  the  following  supporting  comments: 

Effective  Communicator:  “A  leader  needs  to  be  able  to  com¬ 
municate  ideas,  policy,  etc.,  up  and  down  the  chain  for  his 
unit  to  be  effective  and  feel  that  they  are  valued  enough  to  be 
kept  in  the  loop  on  decisions  impacting  them ....  As  a  leader, 
you  need  to  issue  clear  instructions  for  your  subordinates 
to  follow,  as  well  as  easy-to-understand  interpretations  of 
policy  to  enable  your  people  to  follow  them  ....  We  gain  a  lot 
by  lessons  learned  by  following  the  policies  and  procedures 
we  have  in  place,  and  in  order  to  ensure  folks  know  that  they 
exist  or  have  changed,  we  need  to  have  leaders  and  managers 
that  communicate  clearly  and  deliberately.” 

Considering  that  the  importance  of  professional  development,  communi¬ 
cation,  relationships,  and  even  innovation  tend  to  become  more  compelling 
over  time,  the  dichotomy  reinforced  the  importance  of  experience.  With 
more  experience,  supervisors  could  be  learning  later  that  all  four  leadership 
qualities  are  essential  to  their  success. 

For  SLMs  with  15  years  or  more,  the  following  leadership  qualities  fell 
one  standard  deviation  below  the  mean  (x  -  la),  without  any  supporting 
comments: 

Displays  Respect  and  Support  for  Others:  None  given. 

For  SLMs  with  less  than  15  years,  the  following  leadership  qualities  fell 
one  standard  deviation  below  the  mean  (x  -  Ict),  without  any  supporting 
comments: 
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Develops  Self  and  Others,  Effective  Communicator,  Builds 
and  Nurtures  Trust  Relationships,  Displays  Respect  and 
Support  for  Others,  and  Innovator:  None  given. 

Do  the  number  of  years  at  an  organization  like  DAU  influence  the  perceived 
importance  of  certain  leadership  qualities? 

For  SLMs  at  DAU  with  3  to  10  years,  the  following  leadership  quality  rose 
one  standard  deviation  above  the  mean  (x  +  Ict),  with  the  following  support¬ 
ing  comments: 

Critical  Thinking:  “Problem  solving  is  vital ....  will  find  the 
best  path  focused  on  outcomes  and  reality ...  separates  per¬ 
ceptions  and  agendas  from  needs  and  goals.” 

For  SLMs  at  DAU  more  than  10  years,  the  following  leadership  qualities  rose 
one  standard  deviation  above  the  mean  (x  +  la)  as  shown  in  Table  5,  with  the 
following  supporting  comments: 
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Leads  by  Example:  “Cannot  expect  people  to  follow  if  you 
are  not  walking  [the]  talk!  ...  People  are  more  willing  to 
follow  someone  that’s  personally  committed ....  You  have  to 
show  integrity,  show  what  you  expect  of  others,  no  less  than 
what  they  can  expect  of  you  ....  Every  action  a  leader  takes 
is  closely  examined  by  those  he  works  with.  As  a  leader,  you 
broadcast  your  values,  ethics,  competence,  commitment, 
and  knowledge.  These  actions  are  infectious  throughout  the 
organization  and  set  the  standards  for  behavior.” 

Credible:  “A  must  if  you  are  going  to  be  recognized  as  [an] 

SME  in  a  functional  area  within  the  [Defense  Acquisition 
Management]  process  ...  This  gets  to  trust  and  respect .... 
without  which  a  leader  is  inept ...  If  you  aren’t  credible,  you 
could  also  be  regarded  as  insincere,  which  doesn’t  aid  trust 
or  the  internal  organizational  climate,  nor  the  confidence 
of  external  customers  who  count  on  DAU  to  help  develop  a 
professional  acquisition  workforce ....  Most  important  qual¬ 
ity.  Goes  with  integrity.  Without  it,  there  will  be  no  trust.” 

For  MLMs  at  DAU  more  than  10  years,  the  following  leadership  qualities  fell 
one  standard  deviation  below  the  mean  (x  +  Ict)  as  shown  in  Table  5,  with  the 
following  supporting  comments: 

Develops  Self  and  Others:  “A  really  good  manager  seeks  to 
develop  subordinates  to  the  extent  that  they  can  be  given 
‘mission  orders’  to  execute  without  being  given  every  little 
detail  of  how  to  do  it ....  Demonstrates  selflessness,  which 
is  an  important  leadership  quality ...  If  it’s  important  to  the 
supervisor  to  develop  skills  and  education  in  both  them¬ 
selves  and  employees,  it  shows  that  you  care  not  only  about 
the  job,  but  about  making  all  better  at  what  we  do  ....  You 
need  to  encourage  growth  in  your  people  to  fight  against 
stagnation  of  thought  (this  includes  yourself).” 

Displays  Respect  and  Support  for  Others:  “Critical  for  a 
leader.  Treat  others  the  way  you  would  expect  to  be  treated 
....  Respect  is  a  two-way  street.  You  get  what  you  give  .... 
Without  respect  both  ways,  you  have  nothing.  People 
will  only  do  what  they  have  to  in  order  to  get  by;  support 
and  respect  by  the  supervisor  displays  a  trusting  work 
environment.” 
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Change  Agent:  “Having  trust  in  leaders  instills  confidence 
in  them  ....  The  leader’s  credibility  reflects  the  organiza¬ 
tion’s  capabilities.” 

For  SLMs  at  DAU  more  than  10  years,  the  following  leadership  qualities 
fell  one  standard  deviation  below  the  mean  (x  -  la),  with  the  following  sup¬ 
porting  comments: 

Effective  Communicator:  “These  are  all  great  traits  of  a  good 
leader.  Cannot  ‘justify’  why  they  are  at  the  bottom.” 

Competent:  “I  think  it’s  a  component  of  credibility  ....  You 
don’t  have  to  be  the  smartest  guy  or  gal  to  lead,  but  you  have 
to  be  smart  enough  to  surround  yourself  with  the  smart  folks 
and  then  listen  to  them ....  Competent  is  a  minimum  thresh¬ 
old  to  rise  to  a  leadership  position.  Other  attributes  become 
the  delta  between  an  average  leader  and  a  good  leader.” 

Displays  Respect  and  Support  for  Others:  “As  a  leader,  it  is 
important  for  you  to  display  respect  and  support  for  others. 

When  your  followers  recognize  that  you  care  and  respect 
them,  they  will  work  harder  to  accomplish  the  mission.” 

For  MLMs  at  DAU  more  than  10  years,  the  following  leadership  quality 
fell  one  standard  deviation  below  the  mean  (x  -  Ict),  without  any  supporting 
comments: 

Credible,  Promotes  Collaboration  and  Innovation: 

None  given. 


Various  studies  have  shown  that  20  percent  to  67 
percent  of  the  variance  that  measures  the  climate  for 
creativity  in  organizations  is  directly  attributable  to 
leadership  behavior. 
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Various  studies  have  shown  that  20  percent  to  67  percent  of  the  variance 
that  measures  the  climate  for  creativity  in  organizations  is  directly  attribut¬ 
able  to  leadership  behavior.  This  suggests  that  leaders  must  act  in  ways  that 
promote  and  support  organizational  innovation  (Horth  &  Buchner,  2014). 
Over  80  percent  of  executive  leaders  surveyed  in  2007  felt  innovation  was 
a  success  indicator,  although  less  than  30  percent  were  satisfied  with  their 
present  innovation  levels  (Legrand  &  Weiss,  2011).  Coincidentally,  many 
of  the  10  traits  of  innovative  leaders  described  by  Jack  Zenger  and  Joseph 
Folkman  (2014)  in  the  text  that  follows  are  embodied  in  the  top  14  leadership 
qualities  outlined  in  this  study: 

•  Display  excellent  strategic  vision.  The  most  effective  innova¬ 
tion  leaders  could  vividly  describe  their  vision  of  the  future, 
and  as  one  respondent  noted  about  his  boss:  “She  excelled  at 
painting  a  clear  picture  of  the  destination,  while  we  worked  to 
figure  out  how  to  get  there”  (Effective  Communication). 

•  Have  a  strong  customer  focus.  What  was  merely  interesting  to 
the  customer  became  fascinating  to  these  individuals.  They 
sought  to  get  inside  the  customer’s  mind.  They  networked  with 
clients  and  asked  incessant  questions  about  their  needs  and 
wants  (Critical  Thinking). 

•  Create  a  climate  of  reciprocal  trust.  Innovation  often  requires 
some  level  of  risk.  Not  all  innovative  ideas  are  successful. 
These  highly  innovative  leaders  initiated  warm,  collaborative 
relationships  with  the  innovators  who  worked  for  them.  They 
made  themselves  highly  accessible.  Colleagues  knew  that  their 
leader  would  cover  their  backs  and  not  throw  them  under  the 
bus  if  something  went  wrong.  People  were  never  punished  for 
honest  mistakes  (Promotes  Collaboration). 

•  Display  fearless  loyalty  to  doing  what’s  right  for  the  organi¬ 
zation  and  customer.  Pleasing  the  boss  or  some  other  higher 
level  executive  always  took  aback  seat  to  doing  the  right  thing 
for  the  project  or  the  company  (Behavior  Aligns  with  Values). 

•  Put  their  faith  in  a  culture  that  magnifies  upward  communication. 
These  leaders  believed  that  the  best  and  most  innovative  ideas 
bubbled  up  from  underneath.  They  strived  to  create  a  culture 
that  uncorked  good  ideas  from  the  first  level  of  the  organization. 
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They  were  often  described  as  projecting  optimism,  full  of  energy, 
and  always  receptive  to  new  ideas.  Grimness  was  replaced  with 
kidding  and  laughter  (Effective  Communication). 

•  Are  persuasive.  These  individuals  were  highly  effective  in 
getting  others  to  accept  good  ideas.  They  did  not  push  or  force 
their  ideas  onto  their  teams.  Instead,  they  presented  ideas  with 
enthusiasm  and  conviction,  and  the  team  willingly  followed 
(Displays  Respect  and  Support  for  Others). 

•  Excel  at  setting  stretch  goals.  These  goals  required  people  to 
go  far  beyond  just  working  harder.  These  goals  required  that 
they  find  new  ways  to  achieve  a  high  goal  (Critical  Thinking). 

•  Emphasize  speed.  These  leaders  believed  that  speed  scraped  the 
barnacles  off  the  hull  of  the  boat.  Experiments  and  rapid  pro¬ 
totypes  were  preferred  to  lengthy  studies  by  large  committees. 

•  Are  candid  in  their  communication.  These  leaders  were 
described  as  providing  honest,  and  at  times  even  blunt,  feedback. 
Subordinates  felt  they  could  always  count  on  straight  answers 
from  their  leader  (Effective  Communication). 

•  Inspire  and  motivate  through  action.  One  respondent  said,  “For 
innovation  to  exist,  you  have  to  feel  inspired.”  This  comes  from 
a  clear  sense  of  purpose  and  meaning  in  the  work  (Builds  and 
Nurtures  Trust  Relationships). 


Other  Leadership  Qualities 

At  the  end  of  the  survey,  the  respondents  were  also  asked  what  other 
qualities  they  thought  were  important  in  a  DAU  leader  and  why.  Here’s  a 
sampling  of  what  they  had  to  say: 

Ability  to  manage  personnel  issues  effectively.  Problems 
can  quickly  get  out  of  hand  if  not  handled  in  a  quick  and  fair 
manner.  It  will  not  only  impact  the  person,  but  the  percep¬ 
tion  among  others  that  either  you  are  not  handling  it  well 
or  not  handling  it;  thus,  it  may  affect  morale  amongst  the 
other  members. 


144 


Defense  ARJ,  April  SOW,  Vol.  23  No.  2:122-150 


April  2016 


Be  forward  thinking.  In  today’s  world  where  things  change 
constantly,  it  is  important  to  look  ahead  and  try  to  lead  your 
workers  towards  the  more  productive  path;  this  is  part  of 
being  competent;  we  want  to  lead  folks  towards  what  we 
believe  is  the  future  and  not  down  a  dead  end. 

A  leader  should  be  a  good  teacher  and  committed  to  teach¬ 
ing  those  who  work  for  him.  Humility  is  also  an  important 
characteristic. 

Patience  and  persistence.  Bureaucratic  organizations  are 
slow  to  change,  so  leaders  in  DAU  need  to  be  prepared  for 
the  long  haul. 
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Curiosity.  It  is  the  best  antidote  to  complacency. 
Cross-region  collaboration. 

DAU’s  leaders  should  know  their  way  around  the  inside  of 
a  classroom. 


Conclusions 

What  does  all  this  mean?  As  Table  6  shows,  when  it  comes  to  leadership, 
demographic  factors  can  easily  influence  how  individuals  judge  certain  lead¬ 
ership  qualities  through  their  personal  experiences  and  exposure  to  various 
situations.  The  DAU  respondents  who  participated  in  this  particular  study 
highlighted  how  they  fluctuate.  Is  it  a  cause  for  concern?  It  invariably  depends 
on  a  given  scenario  and  what  vital  leadership  qualities  have  either  been  highly 
effective  or  perhaps  marginalized  in  their  view.  Historically,  if  leaders  are 
undervaluing  a  particular  set  of  leadership  imperatives  that  needs  more 
thrust,  it  could  cost  the  organization  they  lead— profoundly.  Polaroid,  Eastman 
Kodak,  Blockbuster,  Eastern  Airlines,  Arthur  Andersen,  DeLorean  Motors, 
Levitz  Furniture,  Enron,  and  many  other  corporations  like  these  learned  what 
happens  when  key  leadership  qualities  lose  all  lift.  These  companies  are  now 
either  resting  in  peace,  have  been  cannibalized  by  another  company,  or  are 
operating  as  a  mere  fraction  of  their  original  size.  Their  leaders  underesti¬ 
mated,  ignored,  and/or  prematurely  dismissed  how  their  culture,  product 
lines,  processes,  corporate  structure,  competition,  customer  base,  outside 
forces,  politics,  etc.,  combined  in  some  way  to  create  a  consequential  nexus 
with  negative  returns.  Their  leaders  had  to  make  hard  choices,  or  tried  to  make 
them  and  subsequently  succumbed  to  insurmountable  organizational 
resistance. 


When  it  conies  to  leadership,  demographic  factors 
can  easily  influence  how  individuals  judge  certain 
leadership  qualities  through  their  personal 
experiences  and  exposure  to  various  situations. 
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On  the  other  hand,  what  leadership  qualities  did  they  discount  too  quickly 
that  would  have  resulted  in  more  favorable  outcomes?  According  to  research, 
leadership  shortcomings  generally  center  on  the  failure  to  recognize  (or 
believe  in)  the  warning  signs  and  respond  in  kind  with  a  confluence  of  these 
same  14  leadership  qualities. 

In  this  study,  the  respondents  had  to  reflect  on  their  experiences  and  decide 
what  still  predominates  today.  The  leadership  qualities  that  rose  to  their 
top  five  were  generally  very  consistent  in  the  aggregate  until  the  slicing 
began.  The  most  significant  fluctuations  occurred  among  four  of  the  top 
five.  Leading  by  Example  saw  much  less  variation.  The  respondents  did  not 
undervalue  any  particular  leadership  quality.  Instead,  they  seemed  to  make 
connections  among  several  below  their  top  five  to  reclaim  their  relative 
importance. 

It’s  difficult  to  attribute  any  one  factor  that  promotes  the  predominance  or 
lessens  some  of  the  leadership  qualities  that  typically  find  their  way  over 
others  in  the  top  five.  This  author  was  particularly  surprised  to  see  where 
Innovator  fell,  however.  Lately,  the  Defense  Acquisition  Executive  and 
his  senior  leaders  have  reinforced  both  its  importance  and  connection  to 
persistence  (Kendall,  2015).  Inarguably,  DAU  is  not  a  technology  company 
and  is  not  necessarily  subject  to  the  same  consequences  of  disruptive  tech¬ 
nology  that  affect  technology  companies.  However,  since  Innovator  fell  so 
markedly  outside  the  respondents’  top  five,  will  it  eventually  result  in  a  neg¬ 
ative  “performance  trajectory”  and  hasten  DAU’s  decline  as  it  did  for  other 
companies  with  the  same  fatal  flight  path  (Christensen,  2015,  pp.  9-21)? 
Even  though  DAU  is  fulfilling  congressional  direction  (in  accordance  with 
the  Defense  Acquisition  Workforce  Improvement  Act,  1991)  to  train  DoD’s 
acquisition  workforce,  many  companies  are  hot  on  its  heels,  vying  to  deliver 
the  same  training  and  other  services  that  DAU  provides.  Because  DAU 
aligns  its  workforce  with  annually  updated  Strategic  Goals  and  measurable 
performance  targets,  this  “development-of-the-fittest”  approach  knowingly 
positions  SLMs  and  MLMs  to  recognize  better,  during  their  development, 
the  early  warning  signs  that  leaders  sometimes  miss— and  sometimes  miss 
too  late  (McCall,  1998,  p.  17).  At  DAU,  the  fluctuations  among  the  14  leader¬ 
ship  qualities  is  no  cause  for  concern  in  this  author’s  opinion.  This  is  so  as 
long  as  SLMs  and  MLMs  who  eventually  take  the  helm  learn  that  both  the 
emphasis  and  relative  importance  of  the  14  leadership  qualities  will  change, 
depending  on  the  nexus  of  all  the  factors  and  conditions  that  could  produce 
real  organizational  peril  if  they  do  not.  And,  that’s  what  matters  the  most. 
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TABLE  6.  RESPONSES  TO  14  LEADERSHIP  QUALITIES  BASED  ON  “AGGREGATE” 


DAU 

Leadership 

Qualities 

Aggregate 

By  Role 

By  Generation 

By  Supervisory  Exp 

By  Years  at  DAU 

X 

Standard 

Deviations  (o) 

Average 

X  ±  la 

SLM  MLM 
AGG  AGG 

SLM 

AD 

SLM 

Faculty 

MLM 

Faculty 

SLM 

Staff 

MLM 

Staff 

SLM 

Boomr 

MLM 

Boomr 

SLM 

Gen 

X 

MLM 

Gen 

X 

SLM 
15  + 

MLM 
15  + 

SLM 
<  15 

MLM 
<  15 

SLM 

3-10 

MLM 

3-10 

SLM 
>  10 

MLM 
>  10 

AVG 

of 

avgs 

ALL  SLM  MLM 

x  +  la 

x  -  la 

Leads  by 
Example 

85%  79% 

75% 

100% 

82% 

90% 

60% 

87% 

77% 

75% 

88% 

86% 

80% 

83% 

78% 

77% 

79% 

93% 

78% 

82% 

9%  8%  7% 

90% 

73% 

Develops  Self  & 
Others 

56%  55% 

67% 

40% 

55% 

50% 

60% 

61% 

57% 

25% 

50% 

67% 

60% 

17% 

50% 

46% 

50% 

64% 

67% 

52% 

14%  17%  6% 

66% 

38% 

Effective 

Communicator 

56%  71% 

67% 

40% 

70% 

50% 

80% 

57% 

67% 

50% 

88% 

57% 

65% 

50% 

78% 

62% 

71% 

50% 

67% 

63% 

13%  7%  7% 

75% 

50% 

Builds  & 
Nurtures  Trust 
Relationships 

44%  47% 

58% 

20% 

42% 

40% 

80% 

52% 

43% 

0% 

63% 

52% 

35% 

17% 

61% 

31% 

42% 

57% 

44% 

43% 

19%  20%  14% 

63% 

24% 

Competent 

37%  45% 

25% 

60% 

42% 

40% 

60% 

30% 

47% 

75% 

38% 

33% 

40% 

50% 

50% 

46% 

54% 

29% 

33% 

44% 

13%  16%  8% 

57% 

31% 

Credible 

37%  21% 

58% 

0% 

24% 

30% 

0% 

39% 

27% 

25% 

0% 

43% 

25% 

17% 

17% 

31% 

29% 

43% 

0% 

24% 

17%  16%  12% 

41% 

7% 

Displays 
Respect  & 
Support  for 
Others 

33%  58% 

17% 

40% 

61% 

50% 

40% 

26% 

60% 

75% 

50% 

24% 

55% 

67% 

61% 

46% 

54% 

21% 

78% 

48% 

18%  20%  10% 

67% 

30% 

Behavior  Aligns 
with  DAU 
Values 

30%  21% 

17% 

60% 

21% 

30% 

20% 

35% 

17% 

0% 

38% 

33% 

15% 

17% 

28% 

23% 

25% 

36% 

22% 

26% 

13%  16%  7% 

39% 

13% 

Critical 

Thinking 

30%  26% 

42% 

20% 

30% 

20% 

0% 

26% 

27% 

50% 

25% 

29% 

35% 

33% 

17% 

38% 

25% 

21% 

22% 

27% 

11%  10%  10% 

38% 

16% 

Exercises 
Authority 
&  Decision 
Making 

26%  26% 

17% 

40% 

27% 

30% 

20% 

17% 

30% 

75% 

13% 

14% 

35% 

67% 

17% 

31% 

21% 

21% 

44% 

31% 

18%  21%  10% 

48% 

13% 

Promotes 

Collaboration 

26%  21% 

8% 

40% 

21% 

40% 

20% 

26% 

20% 

25% 

25% 

24% 

20% 

33% 

22% 

31% 

25% 

21% 

11% 

24% 

8%  9%  4% 

33% 

16% 

Maintains  DAU 
Enterprise 
Perspective 

19%  13% 

17% 

40% 

9% 

10% 

40% 

22% 

13% 

0% 

13% 

19% 

15% 

17% 

11% 

23% 

13% 

14% 

11% 

17% 

10%  10%  9% 

27% 

7% 

Change  Agent 

11%  8% 

8% 

0% 

9% 

20% 

0% 

9% 

10% 

25% 

0% 

5% 

10% 

33% 

6% 

8% 

4% 

14% 

22% 

11% 

9%  10%  7% 

20% 

1% 

Innovator 

11%  8% 

25% 

0% 

6% 

0% 

20% 

13% 

7% 

0% 

13%  14% 

10% 

0% 

6% 

8% 

8% 

14% 

0% 

8% 

7%  8%  5% 

16% 

1% 
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Mehmet  Sahinoglu ,  MSgt  Stephen  Stockton,  USAF  (Ret.), 
Capt  Robert  M.  Barclay,  USAF  (Ret.),  and  Scott  Morton 


Driven  by  the  ubiquity  of  computers  in  modern  life  and  the  subsequent  rise  of 
cybercriminality  and  cyberterrorism  in  the  government  and  defense  industry 
digital  forensics  is  an  increasingly  salient  component  of  the  defense  acquisi¬ 
tion  process.  Though  primarily  located  in  the  law  enforcement  community 
digital  forensics  is  increasingly  practiced  within  the  corporate  world  for  legal 
and  regulatory  requirements.  Digital  forensics  risk  involves  the  assessment, 
acquisition,  and  examination  of  digital  evidence  in  a  manner  that  meets  legal 
standards  of  proof  and  admissibility.  The  authors  adopt  a  model  of  digital 
forensics  risk  assessment  that  quantifies  an  investigator’s  experience  with 


O  /ead  image  by  Diane  Fleischer 


eight  crucial  aspects  of  the  digital  forensics  process.  This  research  adds  the 


concept  of  quantifying  through  a  designed  risk  meter  algorithm  to  calculate 
digital  forensics  risk  indices.  Numerical  and/or  cognitive  data  were  pains¬ 
takingly  collected  to  supply  input  parameters  to  calculate  the  quantitative 
risk  index  for  the  digital  forensics  process.  Much  needed  risk  management 
procedures  and  metrics  are  also  appended. 
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Digital  forensics  is  a  topic  that  has  been  popularized  by  television  pro¬ 
grams  such  as  CSI.  Crime-solving  glamour  and  drama  aside,  the  reality  is 
that  the  digital  forensics  process  is  a  highly  technical  field  that  depends 
on  the  proper  implementation  of  specific,  well-accepted  protocols  and 
procedures.  Inadequate  forensic  tools  and  technical  examination,  as  well 
as  lack  of  adherence  to  appropriate  protocols  and  procedures,  can  result 
in  evidence  that  does  not  meet  legal  standards  of  proof  and  admissibility. 
Digital  forensics  risk  arises,  for  example,  when  personnel  lack  the  proper 
tools  to  conduct  investigations,  fail  to  process  evidentiary  data  properly,  or 
do  not  follow  accepted  protocols  and  procedures. 


Assessing  and  quantifying  digital  forensics  risk  is  the  goal  of  this  article.  To 
do  so,  the  authors  utilize  a  digital  forensics  risk  meter,  based  on  a  series  of 
questions  designed  to  assess  respondents’  perceptions  of  digital  forensics 
risk.  Based  on  the  responses,  a  digital  forensics  risk  index  will  be  calculated. 
Where  this  approach  differs  is  that  other  approaches  typically  provide  gen¬ 
eral  guidance  in  the  form  of  best  practices,  classification  schemes  or,  at  best, 
a  checklist  for  digital  forensics  procedures,  and  do  not  provide  quantitative 
tools  (based  on  game  theory)  for  risk  management  and  mitigation.  Examples 
of  other  such  approaches  follow: 


154 


Defense  ARJ,  April  2016,  Vol.  23  No.  2:152-177 


April  2016 


•  U.S.  Department  of  Justice,  Forensic  Examination  of  Digital 
Evidence:  A  Guide  for  Law  Enforcement  (general  guidelines  and 
worksheets)  (U.S.  Department  of  Justice,  2004) 

•  Error,  Uncertainty,  and  Loss  in  Digital  Evidence  (certainty 
levels)  (Casey,  2002) 

•  Cyber  Criminal  Activity  Analysis  Models  using  Markov  Chain 
for  Digital  Forensics  (suspicion  levels)  (Kim  &  In,  2008) 

•  Two-Dimensional  Evidence  Reliability  Amplification  Process 
Model  for  Digital  Forensics  (evidence  reliability)  (Khatir, 
Hejazi,  &  Sneiders,  2008) 

•  Building  a  Digital  Forensic  Laboratory:  Establishing  and 
Managing  a  Successful  Facility  (checklist)  (Jones  &  Valli,  2011) 

One  approach  that  does  employ  quantification.  Metrics  for  Network  Forensics 
Conviction  Evidence,  is  confined  to  network  forensics— mostly  measuring 
severity  impact— and  does  not  provide  mitigation  advice  (Amran,  Phan, 
&  Parish,  2009).  In  that  research  article,  the  authors  show  “how  security 
metrics  can  be  used  to  sustain  a  sense  of  credibility  to  network  evidence 
gathered  as  an  elaboration  and  extension  to  an  embedded  feature  of  Network 
Forensics  Readiness  (NFR).”  They  then  propose  “a  procedure  of  evidence 
acquisition  in  network  forensics ...  then  analyze  a  sample  of  a  packet  data  in 
order  to  extract  useful  information  as  evidence  through  a  formalized  intu¬ 
itive  model,  based  on  capturing  adversarial  behavior  and  layer  analysis, ... 
apply  the  Common  Vulnerability  Scoring  System— or  CVSS  metrics  to  show 
the  severity  of  network  attacks  committed...”(p.  1). 

The  digital  forensics  risk  meter  presented  in  this  article  will  provide  objec¬ 
tive,  automated,  dollar-based  risk  mitigation  advice  for  interested  parties 
such  as  investigators,  administrators,  and  officers  of  the  court  to  minimize 
digital  forensics  risk.  Figure  1  represents  a  decision  tree  diagram  to  assess 
risk;  Figure  2  (with  the  Advice  column  on  the  right  extracted  from  Figure 
B-l,  Appendix  B)  represents  sample  mitigation  advice  generated  from  the 
respondents’  inputs.  This  article  will  not  only  present  a  quantitative  model, 
but  will  generate  a  prototype  numerical  index  that  facilitates  appropriate 
protocols  and  procedures  to  ensure  that  legal  standards  of  proof  and  admis¬ 
sibility  are  met. 
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FIGURE  1.  DIGITAL  FORENSICS  RISK  DIAGRAM 
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FIGURE  2.  MEDIAN  DIGITAL  FORENSICS  RISK  METER  RESULTS  MITIGATED  TO  35.83%,  CONTINUED 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


O 

u 

> 

5 


5  tn 

.£  O 

il  o 


■s  «i 
=  ° 
3  O 


*8 
O  o 


o 

0) 

c 

(0 


Q  w 

a  a 


oB  z 

z  q 

o  j 


0)  n 

a  a 


°5  Z 

o  -■ 


ru 

0 


12 

rc 

o 

c 

3 

> 


f 0 

0  o3  0 


.52“ 

4_I  i-  -t-* 

>  ro  SS  ro  c 


~  FT  o:  =  5  a:  ld  £  E 

0  >>  f'  0 

"  F  n  ?  F  n  !?  >  'S 

(D  ro  +-*  Chn  i-  ^ 

s-a(JacqcO)Q^ 

^  ro  >  3  :>  °  d  cs 

_u=  co>  =  ^0).Ecn 


K) 

CM 

o 

LD 

ee- 

LO 

O. 

CM 


to  CO  to 

c  o 
|2  il  u 


“  c  *■* 

®  (11  w 

SI  >  o 

m  m  <-> 


U 


81 

«  g 

4-*  C 

o  03 


U 


158 


Defense  ARJ,  April  2016,  Vol.  23  No.  2:152-177 


FIGURE  2.  MEDIAN  DIGITAL  FORENSICS  RISK  METER  RESULTS  MITIGATED  TO  35.83%,  CONTINUED 


April  2016 


(/) 

o 

o 

4-J 

c 

D 

0 

O) 

c 

<D 

-C 

U 


hs 

K) 

hO 

CO 

LO 

hO 


U) 

q: 

"fD 

4— ' 

,o 


(\ 

K) 

hO 

CO 

LO 


tf> 

o' 

-M 

,0 


o 

o 


fD 

u 


U 


Z  -Q 
u  2 

S  c 

I  > 


Defense ARJ,  April  2016,  Vol.  23  No.  2 :  152-177 


159 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Vulnerabilities,  Threats,  and 
Countermeasures 

Based  on  industry  best  practices  guidelines,  such  as  the  U.S.  Department 
of  Justice  (2004)  Forensic  Examination  of  Digital  Evidence:  A  Guide  for  Law 
Enforcement,  eight  specific  vulnerabilities  are  assessed: 

1.  Protocols  and  Procedures 

2.  Evidence  Assessment 

3.  Evidence  Acquisition 

4.  Evidence  Examination 

5.  Documentation  and  Reporting 

6.  Digital  Forensics  Tools 

7.  Legal  Aspects 

8.  Victim  Relations 

Within  each  vulnerability  category,  questions  pertain  to  specific  threats  and 
countermeasures.  For  example,  within  the  Evidence  Acquisition  vulnera¬ 
bility,  respondents  are  asked  questions  regarding  precautions,  protection, 
and  preservation  threats  and  countermeasures.  Within  the  Evidence 
Examination  vulnerability,  respondents  are  asked  questions  regarding 
preparation,  physical  extraction,  logical  extraction,  timeframe  analysis, 
data  hiding  analysis,  application/file  analysis,  and  ownership/possession 
threats  and  countermeasures.  Within  the  digital  forensics  Tools  vulnerabil¬ 
ity,  respondents  are  asked  questions  regarding  hardware,  software,  training, 
and  funding  threats  and  countermeasures.  Figure  1  details  these  vulnera¬ 
bilities  and  threats.  The  responses  are  then  used  to  generate  a  quantitative 
Digital  Forensics  risk  index. 


Assessment  Questions 

Questions  are  designed  to  elicit  responses  regarding  the  perceived  risk 
to  proper  Digital  Forensics  procedures,  evidence  handling/examination, 
admissibility,  and  other  associated  issues  from  particular  threats,  as  well 
as  the  countermeasures  the  respondents  may  employ  to  counteract  those 
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threats.  For  example,  in  the  Evidence  Examination  vulnerability,  questions 
regarding  the  data  hiding  analysis  threat  include  both  threat  and  counter¬ 
measure  questions.  Threat  questions  would  include: 

•  Do  file  headers  not  correspond  to  file  extensions? 

•  Did  the  suspect  encrypt  or  password-protect  data? 

•  Are  hidden  messages  present? 

•  Are  host-protected  areas  (HPA)  present? 


Countermeasure  questions  would  include: 

•  Did  the  examiner  correlate  file  headers  to  the  corresponding 
file  extensions  to  identify  any  mismatches  that  may  indicate 
the  user  intentionally  hid  data? 
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•  Did  the  examiner  gain  access  to  all  password-protected, 
encrypted,  and  compressed  files,  which  may  indicate  an 
attempt  to  conceal  the  data  from  unauthorized  users? 

•  Did  the  examiner  conduct  a  thorough  stenographic  analysis? 

•  Did  the  examiner  gain  access  to  HPAs  that  may  indicate  an 
attempt  to  conceal  data? 

Sample  vulnerability  (Evidence  Acquisition)  assessment  questions 
employed  in  the  digital  forensics  risk  meter  are  found  in  Appendix  A. 
Appendix  A  also  clarifies  and  precludes  confusion  between  Evidence 
Acquisition  and  materiel  acquisition.  The  first  proactive  step  in  any  digi¬ 
tal  forensic  investigation  is  acquisition.  The  inherent  problem  with  digital 
media  is  that  it  is  readily  modified  just  by  accessing  files.  Working  from 
a  copy  is  one  of  the  fundamental  steps  to  making  a  forensic  investigation 
auditable  and  acceptable  to  a  court  (Acquisition,  n.d.). 


Risk  Calculation  and  Risk  Management 
through  Surveys 

Based  on  their  experience,  the  respondents  answer  yes  or  no  to  the 
survey  questions.  These  responses  are  then  used  to  calculate  residual  risk. 
Employing  a  game-theoretical  mathematical  approach,  the  calculated  risk 

index  is  used  to  generate  an  optimization  or  lowering  _ _ 

of  risk  to  desired  levels  (Sahinoglu,  2007,  2016). 

A  more  detailed  set  of  mitigation  advice  will  be 
generated  to  show  interested  parties  (such  as  inves¬ 
tigators,  administrators,  and  officers  of  the  court) 
where  risk  can  be  reduced  to  optimized  or  desired 
levels.  An  example  of  such  risk  reduction  is  shown 
in  Figure  2,  from  45.8  percent  to  35.8  percent, 
which  represents  the  median  response  from  the 
study  participants  (Sahinoglu,  Cueva-Parra,  &  Ang, 

2012).  Figure  2  is  an  actual  screenshot  of  a  results 
table,  representing  the  median  digital  forensics  risk 
meter  results  displaying  threat,  countermeasures, 
residual  risk  indices,  optimization  options,  and 
risk  mitigation  advice.  For  this  study,  a  random 
sample  of  responses  from  27  survey  par¬ 
ticipants  was  analyzed;  their  residual 
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risk  results  are  tabulated  and  presented  in  Appendix  B.  The  survey  portfo¬ 
lio  used  in  this  assessment  and  upon  which  this  research  article  is  based 
showed  the  complexity  of  the  digital  forensics  field,  encompassing  tools, 
procedures,  specific  training,  budget,  and  trial. 

Digital  forensics  has  two  crucial  phases  (Appendix  A).  The  first  phase 
included  all  the  forensics  involved  with  the  collection  of  data,  while  the 
second  phase  concerns  defending  the  data  collected,  the  means  by  which 
the  data  were  collected,  and  chain  of  custody  applied  from  the  original 
collection  until  court  (Sahinoglu,  Stockton,  Morton,  Barclay,  &  Eryilmaz, 
2014).  The  initial  goal  was  to  obtain  survey  input  from  local  city  leaders  in 
Montgomery,  Alabama.  Although  individuals  from  the  Governor’s  Office, 
Montgomery  Police  Department,  and  District  Attorney’s  office  were  will¬ 
ing  to  assist,  our  short  timeframe  and  their  busy  schedules  prevented  their 
offices  from  providing  input  to  the  digital  forensics  survey.  Fortunately,  the 
authors  had  contacts  at  other  law  enforcement  offices,  which  agreed  to  make 
personnel  available  for  the  survey  and  eventual  follow-up.  Eventually,  three 
law  enforcement  offices  and  one  special  investigation/training  organization 
participated  and  provided  valuable  input. 


Our  first  objective  was  to  explain  the  purpose  of  the  survey  and  the  potential 
value  the  combined  results  could  offer  each  of  the  offices.  At  each  location, 
participants  included  investigators,  initial  responders,  digital  forensics 
specialists,  and  legal  experts  (i.e.,  District  Attorney  Office  personnel). 
The  range  of  expertise  of  the  participants  was  invaluable,  as  each  pro¬ 
vided  insight  into  an  aspect  of  the  survey  that  is  often 
unique  to  a  position  within  a  department.  Because 
of  this  range  of  expertise,  the  authors  are  confident 
they  were  able  to  capture  the  three  main  components 
of  the  survey  portion  of  the  Risk-o-Meter  (RoM). 
Perspectives  from  collection  of  evidence,  packaging 
of  evidence  for  trial,  and  presentation  of  evidence  at 
I  trial  were  all  given.  Although  the  special  investiga¬ 
tion/training  organization  had  many  fewer  survey 
participants,  they  did  offer  a  unique  perspective,  as 
they  represented  an  organization  that  focuses  on 
training  digital  forensics  experts  for  the  military. 

The  results  were  then  run  for  each  participant, 
determining  the  Initial  Repair  Cost  to  Mitigate. 
This  was  determined  by  using  a  Criticality 
t  of  1.0,  Equipment  Cost  of  $0.0,  and  a 
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Production  Cost  of  $1,000.  The  median  of  all  results  was  determined  and 
then  optimized  through  the  RoM  to  determine  the  best  “bang  for  the  buck” 
that  would  reduce  the  participant’s  Total  Residual  Risk  by  10  percent.  The 
initial  Total  Residual  Risk  for  the  median  participant  was  45.8  percent,  with 
an  Expected  Cost  of  Loss  (ECL)  of  $458.34.  Once  optimized,  the  Total  Risk 
was  reduced  to  35.8  percent,  and  the  ECL  was  reduced  by  $100  to  a  total 
ECL  of  $358.34  (Figure  2).  The  first  optimized  solution  was  to  increase 
the  countermeasure  (CM)  capacity  for  the  “Examiner  Notes”  threat  for 
the  Documentation  and  Reporting  vulnerability  from  45.0  percent  to  72.17 
percent,  for  an  improvement  of  27.17  percent.  The  second  optimized  solution 
was  to  increase  the  CM  capacity  for  the  “Victim  Rights  and  Support”  threat 
for  the  Victim  Relations  vulnerability  from  72.50  percent  to  99.92  percent, 
for  an  improvement  of 27.42  percent. 
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Table  B-2  in  Appendix  B  depicts 
a  set  of  constrained  linear 
equations  used  within  the  body 
of  the  risk  meter’s  innovative 
second-stage  software  for  the 
game-theoretic  optimization 
necessary  to  create  the  Advice 
column  (shown  on  the  right  in 
Figure  2).  The  Advice  column’s 
original  survey  calculations  are 
depicted  in  Figure  B-l,  which 
displays  company  ECS08: 14th 
Ranked  Overall  Median  Survey. 
This  is  followed  by  Figure  B-2, 
which  displays  company  OPDl’s 
Group  Median  Survey  Taker’s 
Original  Survey  Outcome;  while  Figure  B-3  displays  company  AUPD5’s 
Group  Median  Survey  Taker’s  Original  Survey  Outcome.  In  each  case,  the 
company  representative  seemed  impressed  with  the  results  and  noted  the 
results  for  possible  future  implementation.  One  organization  actually  com¬ 
mented  that  they  had  already  begun  looking  into  increases  in  at  least  one 
CM  that  was  identified  by  the  optimization.  Clearly,  this  episode  validated 
the  tool  and  its  usefulness  in  their  eyes. 
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Discussion  and  Conclusions 

The  advantages  of  conducting  business  on  the  Internet  have  been  well 
documented.  Conducting  business  online  is  frequently  faster  and  cheaper 
than  utilizing  traditional  methods.  However,  this  comes  with  the  digital 
forensics-related  vulnerabilities  and  pertinent  threats  that  tend  to  convert 
the  positive  advantages  to  clear  disadvantages  as  a  result  of  fraud  and 
wrongdoing.  With  the  advent  of  the  Internet  and  burgeoning  information 
systems,  digital  forensics  has  gained  worldwide  momentum.  In  every  envi¬ 
ronment,  the  content  of  digital  information  relative  to  criminal  undertakings 
and  investigations  alike  has  vastly  increased,  growing  disproportionately 
to  the  capacities  of  state  and  local  governments,  as  well  as  federal  agencies 
and  military  components.  The  risk  assessment,  risk  mitigation,  or  general 
risk  management  that  involve  planned  investment  policy  in  order  of  priority, 
with  a  sound  and  auditable,  cost-effective  approach,  are  missing  links.  The 
proposed  digital  forensics  risk  meter  is  an  innovative  initiative  that  provides 
a  quantitative  assessment  of  risk  to  the  user  as  well  as  recommendations 
for  mitigating  that  risk.  This  approach  will  be  a  highly  useful  tool  to  inter¬ 
ested  parties  such  as  investigators,  company  or  system  administrators,  and 
officers  of  the  court  seeking  to  minimize  and  thereby  mitigate  digital  foren¬ 
sics  risk  by  leveraging  and  introducing  early,  preventive  CMs  identified  as 
an  outcome  of  this  dynamic  closed-end  survey. 


This  approach  will  be  a  highly  useful  tool  to  interested 
parties  such  as  investigators,  company  or  system  admin¬ 
istrators,  and  officers  of  the  court  seeking  to  minimize 
and  thereby  mitigate  digital  forensics  risk  by  leveraging 
and  introducing  early,  preventive  CMs  identified  as  an 
outcome  of  this  dynamic  closed-end  survey. 


Additional  future  research  by  the  principal  author  will  involve  the  addition 
of  cloud  computing  concerns  such  as  service  provider  cooperation  and  data 
accessibility,  as  well  as  the  incorporation  of  new  questions  so  as  to  better 
refine  user  responses  and  subsequent  calculation  of  risk  and  mitigation  rec¬ 
ommendations.  Minimization  or  mitigation  of  digital  forensics  risk  will 
greatly  facilitate  the  success  of  digital  forensics  investigations,  ensuring  that 
legal  standards  of  proof  and  admissibility  are  ultimately  met.  The  digital 
forensics  risk  meter  tool  provides  the  means  to  identify  areas  where  risk  can 
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be  minimized,  as  well  as  giving  the  objective,  dollar-based  mitigation  advice 
to  do  just  that.  This  aspect  of  objective  quantifiable  risk  assessment  and  man¬ 
agement  will  add  to  the  trustworthiness  of  acquisition  practices  in  terms  of 
dependable  Internet  communications  involving  great  quantities  of  materiel 
and  their  budgetary  repercussions. 


Limitations  and  Future  Research 

The  limitations  are  obvious  due  to  input  data  deficiency,  but  methods 
such  as  the  one  proposed  in  this  article  are  a  good  way  to  start  due  to  the 
objective,  hands-off,  automated,  cost-effective  treatment  of  the  problem  at 
hand.  Sound  assessment  of  digital  forensics  risk  can  result  when  informa¬ 
tion  entered,  from  learned  respondents,  is  as  close  to  the  truth  as  feasibly 
possible.  The  discussion  that  follows  clarifies  how  this  proposed  work  is 
directly  relevant  to  acquisition  reisk  mitigation  if  applied  appropriately 
within  a  system. 

This  research  article  is  not  focused  on  the  usual  law  enforcement  or  digi¬ 
tal-policing  procedures,  but  is  directed  towards  greater  awareness  for  the 
in-house  (e.g.,  acquisition  community)  workforce  as  they  manage  already 
existing  risk  assessment  and  risk  management  algorithms.  By  leveraging 
the  countermeasures  outlined  in  this  article  (in  particular,  the  Advice  col¬ 
umn  in  Figure  2,  which  employs  probability- estimation  and  game-theoretic 
risk  computing),  the  authors  anticipate  that  acquisition  practitioners  can 
better  preclude  future  digital  forensics  breaches  by  taking  timely  CMs. 

Law  enforcement,  in  cooperation  with  the  defense  acquisition  community, 
is  increasingly  becoming  an  important  player  in  digital  forensics,  thereby 
lending  increased  scrutiny  in  this  vital  area.  Law  enforcement  is  more  aware 
of  evidence  such  as  drug  cartel  activity  and  money  laundering  through  all 
avenues  such  as  export,  import,  and  domestic  acquisition  activities.  Even 
in  homicide  cases,  much  useful  evidence  can  be  deduced  by  using  digital 
forensics  information.  In  addition,  digital  forensics  sciences  not  only  can 
break  a  difficult  case,  but  can  do  so  quickly  and  inexpensively  compared  to 
police  detectives’  usual  time-tested,  but  tedious  practices.  The  proposed 
risk  meter  software  and  its  algorithm  can  successfully  lead  the  way  toward 
navigating  the  stages  of  cost-effective  risk  assessment  and  management. 

In  conclusion,  the  best  “bang  for  the  buck”  derives  from  simple  usability 
and  scientific  objectivity. 
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Appendix  A 

Sample  Vulnerability  (Evidence  Acquisition,  Documentation 
and  Reporting,  and  Victim  Relations)  Assessment  Questions 
(in  XML  format)  and  Survey  Template 

<survey> 

Vulnerability  title=  “Evidence  Acquisition”  level=  “0”> 

<vQuestion>  Are  special  precautions  not  taken  to  preserve  digital  evidence? 
</vQuestion> 

<vQuestion>  Was  write  protection  not  utilized  to  preserve  and  protect 
original  evidence?  </vQuestion> 

<vQuestion>  Was  digital  evidence  not  secured  in  accordance  with 
departmental  guidelines?  </vQuestion> 

<vQuestion>  Was  speed  the  primary  concern  when  it  came  to  acquiring 
digital  evidence?  </vQuestion> 

cthreat  title  =  “Precautions’^ 

<tQuestion>  Was  evidence  on  storage  devices  destroyed  or  altered? 
</tQuestion> 

<tQuestion>  Was  equipment  damaged  by  static  electricity  and  magnetic 
fields?  </tQuestion> 

<tQuestion>  Was  the  original  internal  configuration  of  storage  devices  and 
hardware  unnoted?  </tQuestion> 

<tQuestion>  Were  investigators  unable  to  provide  drive  attributes? 
</tQuestion> 

<threat  title  =  “Protection’^ 

<tQuestion>  Was  CMOS/BIOS  information  not  captured?  </tQuestion> 
<tQuestion>  Was  the  computer’s  functionality  and  the  forensic  boot  disk 
not  tested?  </tQuestion> 

<tQuestion>  Did  the  forensic  boot  disk  not  boot?  </tQuestion> 

<tQuestion>  Did  the  investigators  not  collect  drive  configuration  information 
from  the  CMOS/BIOS?  </tQuestion> 


<threat  title  =  “Preservation’^ 

<tQuestion>  Did  the  investigators  not  perform  the  acquisition  using  the 
examiner’s  system?  </tQuestion> 

<tQuestion>  Was  a  RAID  present  in  the  subject  system?  </tQuestion> 
<tQuestion>  Was  host-specific  data  not  captured?  </tQuestion> 
<tQuestion>  Was  successful  acquisition  not  verified?  </tQuestion> 


</threat> 

</vulnerability> 

</survey 
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DIGITAL  FORENSICS  RISK  SURVEY 


This  survey  has  8  main  categories  of  vulnerabilities.  Please  identify  the 
areas  below  where  you  have  observed  vulnerabilities  while  involved  with 
digital  forensics  activities  within  your  organization. 


*  A  minimum  of  2  categories  must  be  chosen: 


Vulnerability  Area 

Reference  Page 

□ 

Protocols  &  Procedures 

Pages  1  &  2 

□ 

Evidence  Assessment 

Pages  3  &  4 

□ 

Evidence  Acquisition 

Page  5 

□ 

Evidence  Examination 

Pages  6  &  7 

□ 

Documentation  &  Reporting 

Page  8 

□ 

Digital  Forensics  Tools 

Page  9 

□ 

Legal  Aspects 

Page  10 

□ 

Victim  Relations 

Page  11 

DIRECTIONS: 

This  Page: 

•  Select  all  vulnerability  areas  that  apply 

•  Proceed  to  appropriate  pages  to  complete  survey  for  each  vulnerability 
area 

Survey  Page(s): 

Vulnerability 

•  Rate  Vulnerability  (0.1-10)  with  10  being  most  vulnerable  and  0.1  being 
least  vulnerable 

•  Select  all  vulnerability  statements  that  apply  ( must  choose  at  least  one ) 

Threat 

•  Rate  Threat  (0.1-10)  with  10  being  greatest  threat  and  0.1  being  the  least 
threat 

•  Using  square  check  box,  select  all  threat  statements  that  apply  to  each 
threat  category  chosen  ( must  choose  at  least  one ) 

Countermeasure 

•  Rate  associated  Countermeasure  for  each  threat  category  chosen  above 
(0.1-10)  with  0.1  being  least  effective  and  10  being  the  most  effective 
countermeasure 

•  Using  square  check  box,  select  all  countermeasure  statements  that 
apply  (must  choose  at  least  one ) 
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Rate  (01.-10)  if  vulnerability  applies 


Vulnerability:  Legal  Aspects 


□  Legal  authority  for  forensic  examinations 
is  unclear 

Must  select  one  (minimum)  for 
each  Vulnerability  selected 

□  The  extent  of  the  authority  to  search  is 
unstated  ^ 

Rate  (0.1-10)  for  all  Threats 

□  Courtroom  admissibility  is  not  a  prime 
consideration  , 

that  apply  "v 

Threat:  Jurisdiction 


Countermeasures 


□  There  is  conflicting  jurisdiction  □ 

Jurisdiction  is  established  among 
agencies  prior  to  investigations 

□  Multiple  jurisdictions  are  often  involved  □ 

Investigators  and  other  officials  from 
different  areas  coordinate  and  cooperate 
on  cases 

□  Potential  evidentiary  data  are  stored  on  □ 

the  cloud  or  some  other  distant  network 

resource 

Court  orders  are  obtained  when  requiring 
distant  service  providers  to  provide 
potentially  evidentiary  data 

□  Cases  often  cross  international  borders  □ 

There  are  bilateral  or  multilateral 
agreements  that  facilitate  cooperation 
with  foreign  law  enforcement  agencies 

Countermeasures 


Threat:  Search  &  Seizure 


□  Cases  are  often  challenged  for  lack  of  □ 

probable  cause 

Forensic  investigators  unequivocally 
identify  and  articulate  a  probable  cause 
necessary  to  obtain  search  warrants 

□  On-site  investigators  often  proceed  □ 

without  knowledge  of  a  warrant 

Search  warrants  are  obtained  prior  to 
investigation  on  site 

□  Investigators  go  beyond  warrants  □ 

originally  used  to  assert  search  authority 

New  search  warrants  are  obtained  as  new 
evidence  is  uncovered  to  avoid  charges 
of  “stale”  warrants 

□  The  evidentiary  chain  of  custody  is  often  □ 

challenged 

Full  documentation  of  the  evidentiary 
chain  of  custody  is  maintained 
throughout  the  investigation 

Threat:  Admissibility 


Countermeasures 


□  Digital  evidence  is  sometimes  changed  □ 

by  seizure 

Strict  measures  are  taken  to  ensure  that 
when  seizing  digital  evidence,  the  action 
does  not  change  that  evidence 

□  Individuals  besides  forensic  investigators  □ 

access  original  digital  evidence 

Only  forensically  competent  persons 
are  allowed  access  to  original  digital 
evidence 

□  Does  activity  related  to  cases  come  □ 

under  legal/judicial  review 

All  activities  related  to  seizures,  access, 
storage,  or  transfer  of  digital  evidence 
are  fully  documented,  preserved,  and 
available  for  legal/judicial  review 

□  The  state  of  evidence  is  often  unknown  □ 

prior  to  opening  files 

Evidence  is  “frozen”  prior  to  opening  the 
files 

Must  select  one  (minimum)  Threat  for  each  vulnerability  selected 
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Appendix  B 

Respondent  Results  Tabulations 


TABLE  B-1.  COMPANIES’/RESPONDENTS’  (AFIT,  AUPD,  ECSO,  OPD) 
SURVEY  RESULTS  FOR  DIGITAL  FORENSICS  RISK  METER  STUDY 

Survey 

Taker 

Residual 
Risk  % 

Ranked  Overall 
(Out  of  27) 

Remarks 

AFIT1 

52.47 

6th 

2nd  out  of  4  within  AFIT 

AFIT2 

49.90 

9th 

3rd  out  of  4  within  AFIT 

AFIT3 

52.71 

5th 

1st  out  of  4  within  AFIT 

AFIT4 

47.64 

10th 

4th  out  of  4  within  AFIT 

AUPD1 

31.15 

26th 

7th  out  of  7  within  AUPD 

AUPD2 

39.67 

20th 

5th  out  of  7  within  AUPD 

AUPD3 

50.02 

8th 

1st  out  of  7  within  AUPD 

AUPD4 

36.98 

21st 

6th  out  of  7  within  AUPD 

AUPD5 

44.59 

16th  -  Overall  Average 

4th  out  of  7  within  AUPD 

AUPD6 

46.06 

13th 

3rd  out  of  7  within  AUPD 

AUPD7 

47.06 

11th 

2nd  out  of  7  within  AUPD 

ECSOl 

51.80 

7th 

5th  out  of  9  within  ECSO 

ECS02 

46.66 

12th 

6th  out  of  9  within  ECSO 

ECS03 

56.94 

2nd 

2nd  out  of  9  within  ECSO 

ECS04 

57.67 

1st 

1st  out  of  9  within  ECSO 

ECS05 

54.87 

3rd 

3rd  out  of  9  within  ECSO 

ECS06 

41.36 

19th 

9th  out  of  9  within  ECSO 

ECS07 

54.84 

4th 

4th  out  of  9  within  ECSO 

ECS08 

45.83 

14th  Overall  Average 

7th  out  of  9  within  ECSO 

ECS09 

45.01 

15th 

8th  out  of  9  within  ECSO 

OPD1 

35.00 

23rd 

4th  out  of  7  within  OPD 

OPD2 

42.56 

18th 

2nd  out  of  7  within  OPD 

OPD3 

44.35 

17th 

1st  out  of  7  within  OPD 

OPD4 

33.39 

25th 

6th  out  of  7  within  OPD 

OPD5 

28.23 

27th 

7th  out  of  7  within  OPD 

OPD6 

34.39 

24th 

5th  out  of  7  within  OPD 

OPD7 

36.41 

22nd 

3rd  out  of  7  within  OPD 

Note.  Respondents  are  ranked  within  and  overall,  where  Median  is  45.83%  (ECS08)  and 
Average  is  44.73%  (AUPD5:  44.49%  is  the  closest  respondent  to  44.7%). 
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TABLE  B-2.  SET  OF  CONSTRAINED  LINEAR  EQUATIONS 
FOR  TABLE  B-1’S  MEDIAN 


Min  COLLOSS  (Column  loss),  s.  t.  (subject  to): 

CM,,  <  1  (1),  CM,2  <  1  (2),  CM, 3  <  1  (3),  CM2,  <1  (4),  CM22  <1  (5),  CM3,  <1  (6), 

CM32  <1  (7),  CM33  <1  (8),  COLLOSS  <1  (9) 

CM,,  >  0.675  (10),  CM,2  >  0.475  (11),  CM,3  >  0.725  (12), 

CM2,  >  0.725  (13),  CM22  >  0.725  (14), 

CM3,  >  0.675  (15),  CM32  >  0.675  (16),  CM33  >  0.675  (17), 

0.09148  CM,,  -1COLLOSS  <  0  (18),  0.05231  CM,2  -1COLLOSS  <  0  (19), 

0.07629  CM, 3  -1COLLOSS  <  0  (20),  0.17734  CM2,  -1COLLOSS  <  0  (21), 

0.13966  CM22  -1COLLOSS  <  0  (22),  0.18896  CM3,  -  1COLLOSS  <  0  (23), 

0.11601  CM32  -1COLLOSS  <  0  (24),  0.15787  CM33  -1COLLOSS  <  0  (25), 

0.09148  CM,,  +  0.05231  CM,2  +  0.07629  CM,3  +  0.17734  CM2,  +  0.13966  CM22  + 
0.18896  CM3,  +  0.11601  CM32  +  0.15787  CM33  >  1-  0.3583  =  1-0.3583  =  0.6417  (26) 


Note.  Used  to  attain  a  risk  mitigated  to  35.83%  from  an  undesirable  45.83%  inspired  by 
Figure  2;  where  Total  #  Constraints  =  3  *  #Selected  Threats  +  2  =  3*8  +  2  =  24  +  2  =  26 
along  with  Objective(Min). 
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Catalysts  of  Military  Innovation 


A  Case  Study  of  Defense 


BIOMETRICS 


COL  Glenn  Voelz,  USA 


Military  innovation  is  a  central  component  of 
U.S.  strategic  advantage;  however,  the  precise  condi¬ 
tions  that  enable  such  innovation  remain  a  matter  of 
debate.  The  recent  introduction  of  biometrics  onto  the 
battlefield  offers  a  useful  case  study  for  examining  catalysts 
of  military  innovation  and  specific  factors  that  enabled  the 
Department  of  Defense  to  rapidly  field  new  technologies  in 
response  to  urgent  operational  requirements.  This  article 
considers  how  doctrinal  design  and  warfighting  strategies 
became  important  catalysts,  and  how  challenges  associated 
with  rapid  fielding,  interoperability,  and  training  limited 
U.S.  forces  from  realizing  the  full  potential  of  these  new 
technologies.  This  case  study  proposes  that  military  inno¬ 
vation  can  occur  only  by  using  an  integrated  approach  that 
encompasses  the  interdependent  elements  of  technology, 
acquisition,  doctrinal  design,  and  warfighting  strategies.  It 
offers  general  conclusions  on  conditions  that  create  fertile 
environments  for  military  innovation  and  identifies  lessons 
learned  for  future  efforts  at  introducing  new  technologies 
into  the  field. 


Keywords:  Doctrine,  acquisition  strategy,  Department  of  Defense 
Automated  Biometric  Identification  System 
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Military  innovation  has  reemerged  as  atopic  of  interest  among  national 
security  professionals.  This  has  been  spurred  by  a  growing  concern  that  the 
United  States  has  ceded  the  military-technological  advantage  it  enjoyed 
for  most  of  the  post-World  War  II  era.  The  push  to  regain  this  edge  has  led 
to  a  number  of  new  initiatives  such  as  Better  Buying  Power  3.0,  aimed  at 
accelerating  acquisition  reform  and  incentivizing  innovation  within  govern¬ 
ment.  Similarly,  the  Department  of  Defense  (DoD)  recently  announced  the 
Defense  Innovation  Initiative,  a  set  of  long-range  research  and  development 
programs  intended  to  identify  advanced  capabilities  as  the  basis  of  a  “Third 
Offset  Strategy.”  These  efforts  focus  on  achieving  high-payoff  breakthroughs 
in  areas  such  as  artificial  intelligence,  robotics,  additive  manufacturing, 
and  nanotechnology,  among  others.  Last  year,  Secretary  of  Defense  Ashton 
Carter  opened  the  Defense  Innovation  Unit  Experimental  in  Silicon  Valley 
to  “scout,  connect,  and  support  the  innovation  of  disruptive  technology” 
with  potential  military  value.  The  common  theme  among  these  initiatives 
is  to  create  U.S.  strategic  advantage  by  improving  the  process  of  military 
innovation;  however,  the  precise  conditions  that  enable  this  to  occur  remain 
a  matter  of  some  debate. 


One  source  of  insight  comes  from  analyzing  recent  examples  of  military 
innovation  that  emerged  during  the  conflicts  in  Iraq  and  Afghanistan. 
Among  these,  biometrics  offers  a  useful  case  study  of  a  technology  that 
was  virtually  unknown  on  the  battlefield  prior  to  9/11,  yet  by  the  end  of 
the  decade  had  become  a  ubiquitous  feature  of 
U.S.  military  operations.  This  particular 
example  is  instructive  because  it  involved 
the  rapid  and  relatively  successful  integra¬ 
tion  of  a  new  technology  that  substantively 
changed  the  way  U.S.  forces  conducted 
operations  on  the  ground.  This  outcome  was 
due  to  several  factors.  As  an  untested  military 
technology,  biometrics  evolved  concurrently 
with  new  doctrinal  concepts  describing  how 
the  tools  would  be  used  on  the  battlefield  to 
create  desired  effects.  These  capabilities  were 
then  applied  as  part  of  a  coherent  warfighting 
strategy  focused  on  specific  operational  chal¬ 
lenges  encountered  in  Iraq  and  Afghanistan. 

Within  this  context,  biometrics  became  a  key 
enabling  technology  of  population-centric  coun¬ 
terinsurgency,  applied  across  a  range  of  use  cases 
such  as  detainee  management,  high-value  targeting,  and 
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support  to  Rule  of  Law  operations.  However,  despite  the  success  in  rapidly 
moving  these  new  technologies  into  the  field,  in  some  cases  the  operational 
impact  was  limited  due  to  challenges  with  interoperability,  informational 
sharing,  and  training.  The  case  study  of  biometrics  demonstrates  that  effec¬ 
tive  military  innovation  can  only  occur  through  an  integrated  approach  that 
takes  into  account  the  interdependent  elements  of  technology  development, 
acquisition  planning,  doctrinal  design,  and  warfighting  strategy. 


Defining  Military  Innovation 

Innovation  describes  the  process  by  which  a  new  idea,  technology,  or 
method  provides  an  improved  capability  for  addressing  an  existing  need. 
Generally,  it  follows  a  process  of  discovery,  application,  and  exploitation 
where  basic  research  is  transformed  from  a  concept  into  a  tool  or  process 
that  delivers  some  kind  of  operational  advantage.  Scholars  of  military  inno¬ 
vation  look  to  several  characteristics  for  evidence  of  meaningful  change. 
The  first  is  whether  the  process  of  innovation  substantively  alters  the  man¬ 
ner  in  which  military  formations  function  in  the  field.  A  second  factor  is 
whether  these  changes  are  significant  in  terms  of  scope  and  organizational 
impact.  A  third  component  takes  into  account  whether  these  changes  ulti¬ 
mately  produce  greater  military  effectiveness  (Grissom,  2006). 


There  exists  a  relatively  deep  body  of  academic  literature  on 
military  innovation,  examining  the  technological,  cul¬ 
tural,  and  bureaucratic  aspects  of  change.  Much  of  this 
research  focuses  on  “innovation  inhibitors”  that  under¬ 
mine  the  successful  adoption  of  new  technologies  and 
methods  (Jungdahl  &  Macdonald,  2014).  Many  of 
these  studies  apply  the  lens  of  organizational  the¬ 
ory  with  emphasis  on  institutional  factors  such  as 
bureaucratic  culture  and  leadership  dynamics  as 
key  variables  in  the  process  of  innovation  (Avant, 
1994;  Posen,  1984).  Williamson  Murray’s  influen¬ 
tial  study,  Military  Adaptation  in  War,  notes  how 
modern  bureaucratic  and  military  cultures  have 
become  antithetical  to  adaptation,  often  for 
reasons  relating  to  parochial  interests 
or  avoidance  of  negative  consequences 
resulting  from  incorrect  decisions 
(Murray,  2009). 
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Some  experts  consider  wartime  innovation  a  phenomenon  that  must  be 
examined  separately  from  that  of  peacetime  change  (Rosen,  1991).  Indeed, 
with  many  examples  of  wartime  innovation,  the  causal  pathways  of  change 
tend  to  be  somewhat  less  complex  and  highly  responsive  to  the  exigent 
demands  of  the  battlefield.  In  such  instances,  the  act  of  warfighting  becomes 
a  laboratory  for  conducting  “natural  experiments”  in  which  technology 
requirements  are  explicitly  articulated  in  response  to  challenges  posed  by 
an  actual  adversary  rather  than  a  hypothetical  one.  This  situation  provides 
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immediate  tactical  feedback  and  creates  a  powerful  dynamic  for  iterative 
design  and  process  improvement.  These  factors  inevitably  sharpen  how 
operational  needs  are  defined,  while  at  the  same  time  accelerating  the 
bureaucratic  process  of  research,  development,  prototyping,  and  fielding. 

Yet,  even  in  cases  where  explicit  tactical  demands  drive  the  adoption  of  a 
new  military  technology,  these  tools  do  not  exist  in  isolation.  Successful 
diffusion  of  new  technologies  or  methods  still  requires  a  conceptual  driver 
to  guide  the  course  of  innovation.  This  provides  the  context  for  how  a  given 
technology  will  be  employed  on  the  battlefield,  thereby  creating  meaningful 
military  effects.  Importantly,  Williamson  Murray  observes  that  technolog¬ 
ical  sophistication  is  not  necessarily  the  most  critical  factor  of  successful 
innovation.  Rather,  it  is  how  well  a  new  technology  is  incorporated  into  an 
effective  concept  for  fighting  that  matters.  This  emerges  from  evolutionary 
problem  solving  focused  on  specific  operational  challenges.  However,  effec¬ 
tive  implementation  also  requires  a  coherent  framework  of  employment 
grounded  in  doctrine,  operational  concepts,  and  an  overarching  strategic 
vision  for  how  the  technology  will  be  used. 

In  the  case  of  biometrics,  the  key  conceptual  driver  was  the  realization 
that  counterinsurgency  and  counterterrorism  operations  against  irregular 
adversaries  required  different  doctrinal  approaches  and  technical  tools 
than  those  optimized  for  conventional  military  conflict.  In  particular,  the 
intelligence,  surveillance,  and  reconnaissance  technologies  needed  for 
identifying  and  targeting  individual  combatants  and  their  networks  were 
not  the  same  as  those  designed  for  detecting  and  destroying  motorized  rifle 
battalions.  This  new  mode  of  warfare  turned  combatant  identity  into  a  crit¬ 
ical  technical  signature  of  the  battlefield.  In  this  complex  human  terrain, 
biometric  technologies  helped  put  a  uniform  on  the  nation’s  enemies  and 
reduced  their  ability  to  leverage  anonymity  for  military  advantage.  This 
paradigm  shift  in  thinking  about  identity  and  military  targeting  established 
a  clear  operational  role  for  biometrics.  It  firmly  placed  the  new  technology 
within  an  explicit  doctrinal  framework  and  described  how  it  would  be  used 
to  support  the  overarching  warfighting  strategy.  In  the  case  of  biometrics, 
several  specific  factors  were  instrumental  as  catalysts  for  innovation: 

1.  Clear  Operational  Use  Case.  Military  innovation  is  most  effec¬ 
tive  when  it  addresses  a  well-defined  operational  challenge.  As  a 
largely  untested  battlefield  technology,  biometrics  evolved  rapidly 
for  the  simple  reason  that  it  provided  a  practical  solution  to  help 
identify,  track,  and  target  irregular  combatants  fighting  without 
uniforms  or  conventional  formations.  Within  the  context  of  waging 
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counterinsurgency,  biometrics  technologies  offered  a  powerful  tool 
with  a  wide  variety  of  use  cases  such  as  detainee  management,  high- 
value  targeting,  and  support  to  Rule  of  Law  operations. 

2.  Value  Proposition  Linked  to  Doctrinal  and  Strategic 
Concepts.  New  military  technologies  require  a  coherent  concept 
of  employment  that  clearly  demonstrates  their  value  within  a  larger 
doctrinal  and  strategic  framework.  Biometrics  succeeded  in  part 
because  it  was  introduced  within  the  context  of  new  doctrinal  and 
strategic  approaches  focused  on  population-centric  counterinsur¬ 
gency  and  identity-based  targeting.  These  priorities  emerged  within 
the  broader  context  of  Iraq  and  Afghanistan,  where  biometrics 
became  an  increasingly  important  technical  tool  for  navigating 
complex  human  terrain  and  assisting  U.S.  forces  in  waging  war 
against  the  enemy. 

3.  Effective  Bureaucratic  Constituencies.  Military  innovation 
ultimately  occurs  within  an  organizational  context;  therefore, 
it  requires  strong  bureaucratic  advocates  with  the  institutional 
capacity  to  manage  the  development  and  integration  of  new  tech¬ 
nologies.  Biometrics  had  a  distinct  advantage  of  being  a  multiuse 
technology  with  a  broad  range  of  operational  applications.  Just  as 
biometrics  appeared  on  the  battlefield,  the  value  of  the  technology 
was  also  recognized  by  law  enforcement,  Homeland  Security,  and 
the  Intelligence  Community,  thereby  creating  a  critical  mass  of 
interest  groups— all  pushing  for  new  investments.  However,  numer¬ 
ous  constituencies  pursuing  parallel  development  programs  also 
created  challenges  for  interoperability  and  data  sharing  as  the  new 
technologies  evolved. 

4.  Development  Partners  in  a  Competitive  Marketplace.  Military 
innovation  works  best  when  government  works  collaboratively 
with  a  diverse  range  of  development  partners  in  a  dynamic  and 
competitive  marketplace.  As  biometrics  technologies  appeared  on 
the  battlefield,  a  growing  demand  also  emerged  for  new  commercial 
applications  that  drove  a  period  of  rapid  innovations  in  the  nonde¬ 
fense  sector.  This  enabled  DoD  to  benefit  from  significant  private 
investment  in  research,  development,  and  prototyping.  While  DoD 
was  not  the  only  market  driver  of  this  innovation,  it  was  in  a  unique 
position  to  exploit  the  latest  developments  for  the  commercial  sector 
and  adapt  these  tools  directly  to  military  needs. 
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Biometrics  Fundamentals 


As  a  general  term,  biometrics  describes  the  measure  of  biological  and/ 
or  behavioral  characteristics  that  can  be  used  for  automated  recogni¬ 
tion  or  identity  verification.  A  biometric  modality  refers  to  a  type  or 
class  of  biometric  samples  such  as  those  derived  from  a  facial  image, 
fingerprint,  iris,  or  voice  pattern.  Biometric  matching  describes  the  ca¬ 
pability  and/or  process  of  comparing  biometric  data  in  order  to  link 
previously  obtained  biometrics  and  related  contextual  data  to  a  partic¬ 
ular  identity  or  for  the  verification  of  identity  (Defense  Forensics  and 
Biometrics  Agency,  2013).  Biometric  data  can  be  combined  with  bi¬ 
ographical  and  other  contextual  information  to  build  a  "pattern  of  life” 
profile  for  individual  subjects.  When  analyzed  together  with  other  bio¬ 
metric  records  and  all-source  intelligence,  this  information  can  reveal 
connections  among  individuals,  correlate  their  activities,  and  expose 
the  structure  of  their  networks. 


Biometrics  as  Military  Innovation 

One  of  the  early  lessons  learned  from  the  conflicts  in  Iraq  and 
Afghanistan  was  that  many  of  the  legacy  intelligence  technologies  devel¬ 
oped  for  conventional  warfare  against  state-based  adversaries  did  not 
provide  the  kind  of  information  needed  to  effectively  support  counterin¬ 
surgency  operations  and,  in  particular,  identity-based,  high-value  targeting 
(Defense  Science  Board,  2011).  As  the  United  States  shifted  towards  a 
counterinsurgency  strategy,  it  required  population-centric  information 
and  refined  targeting  intelligence  for  identifying,  isolating,  and  eliminating 
insurgents  from  the  battlefield.  These  operational  challenges  demanded 
new  technologies  to  enable  U.S.  forces  to  detect  and  identify  individual 
actors,  characterize  and  geo-locate  their  activities,  and  understand  the 
structure  and  function  of  their  networks.  This  presented  an  enormous 
tactical  dilemma  for  soldiers  fighting  on  an  irregular  battlefield  against 
adversaries  who  did  not  wear  uniforms  and  could  not  easily  be  distin¬ 
guished  from  the  local  population.  As  such,  identity  verification  emerged 
as  one  of  the  major  technical  challenges  of  the  campaigns  in  Iraq  and 
Afghanistan.  Although  relatively  untested  as  a  military  technology,  biomet¬ 
rics  rapidly  emerged  as  an  important  tool  for  differentiating  actors  within 
a  complex  and  often  ambiguous  operational  environment. 

Prior  to  2001,  the  U.S.  military  had  no  significant  operational  experience  in 
the  use  of  biometrics.  DoD’s  original  vision  for  biometrics  was  relatively  lim¬ 
ited  in  scope  and  focused  principally  on  tasks  such  as  information  assurance 
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for  automation  systems  and  physical  access  control  (Defense  Science  Board, 
2007;  National  Science  and  Technology  Council,  2008,  p.  21).  However,  new 
Homeland  Security  concerns  following  9/11  and  the  subsequent  conflicts 
in  Iraq  and  Afghanistan  became  the  initial  catalysts  that  transformed 
biometrics  into  an  operationally  focused  technology.  Although  the  Army’s 
biometric  development  program  had  been  operating  since  1999,  it  was  not 
until  2001  that  the  Battle  Command  Battle  Laboratory  produced  the  first 
Biometric  Automated  Toolset  (BAT)  prototype,  a  multimodal  (fingerprint, 
iris,  and  face)  system  for  collecting,  matching,  and  storing  personally  identi¬ 
fying  information.  This  technology  was  initially  field-tested  in  the  Balkans 
where  it  was  primarily  used  for  identifying  local  national  workers  accessing 
U.S.  installations.  As  these  technologies  matured  from  prototype  design  into 
a  functional  capability,  a  number  of  new  uses  evolved  that  greatly  expanded 
the  value  of  these  tools  across  the  range  of  military  operations. 


Biometrics  Use  Case:  Detainee  Management 


Almost  immediately  at  the  start  of  operations  in  Iraq  and  Afghanistan, 
U.S.  forces  faced  an  unprecedented  challenge  of  managing  the  large 
numbers  of  detainees  on  the  battlefield.  One  report  from  early  in  the 
conflicts  noted  how  the  “handling  of  detainees,  appropriately  docu¬ 
menting  their  capture,  and  identifying  and  accounting  for  them,  were 
all  dysfunctional  processes,  using  little  or  no  automation  tools”  (Jones, 
2004,  p.  21).  New  biometrics  technologies  offered  one  solution  for  this 
dilemma.  In  early  2002,  a  BAT  prototype  was  fielded  to  Joint  Special 
Operations  Command  in  Afghanistan  and  first  used  for  enrolling  per¬ 
sons  of  interest  detained  on  the  battlefield.  By  2003,  similar  systems 
were  deployed  at  detention  facilities  in  Iraq  for  detainee  management 
and  later  as  a  tool  for  generating  biometrically  enhanced  interroga¬ 
tion  reporting  (lasso,  2013).  By  2004,  DoD  directed  that  all  U.S.  mil¬ 
itary  units  worldwide  would  collect  biometric  data  from  detainees 
(DoD,  2004).  One  vivid  demonstration  of  the  value  of  this  data  came 
in  2011  when  500  Taliban  prisoners  escaped  from  Kandahar’s  Sarposa 
prison.  All  detainees  had  previously  undergone  biometric  enrollment, 
and  within  1  month  30  individuals  were  recaptured  in  the  local  area  as 
a  result  of  random  biometric  checks  (The  Eyes  Have  It:  Biometrics  in 
Afghanistan,  2012).  Since  then,  biometric  data  gathered  by  DoD  and 
other  government  agencies  have  been  used  to  identify  and  prevent 
tens  of  thousands  of  potentially  threatening  individuals  from  entering 
the  United  States  (Partnership  for  Public  Service,  2013,  pp.  12-13). 
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The  first  major  operational  employment  of  the  BAT  system  was  by  Marine 
Corps  units  during  the  resettlement  of  Fallujah  following  major  combat 
operations  in  2004.  Handheld  biometric  devices  and  databases  were  used 
to  monitor  the  flow  of  residents  into  and  out  of  the  city  as  a  means  of  identi¬ 
fying  insurgents  moving  among  the  population  (McWilliams  &  Schlosser, 
2014,  p.  62;  Shanker,  2011).  The  use  of  this  technology  on  the  battlefield 
expanded  rapidly  as  the  United  States  shifted  towards  a  population-cen¬ 
tric  counterinsurgency  strategy  in  Iraq  and  became  a  critical  tool  during 
the  “surge”  period  for  identifying  and  segregating  insurgents  from  the 
larger  population.  By  that  time,  thousands  of  BAT  toolsets  and  the  newer 
Handheld  Interagency  Identity  Detection  Equipment  (HIIDE)  systems  had 
been  fielded  to  tactical  units.  Multimodal  or  13-point  biometric  collection 
(10  fingers,  two  irises,  and  one  face)  became  a  standard  feature  of  combat 
patrols  and  documenting  encounters  with  persons  of  interest.  By  the  end  of 
combat  operations  in  Iraq,  U.S.  forces  had  compiled  a  biometric  database 
containing  some  three  million  individual  files  (Ackerman,  2011). 

Biometric  technologies  proved  equally  important  in  Afghanistan  where  few 
inhabitants  possessed  verifiable  identity  documentation  and  combatants 
could  not  easily  be  distinguished  from  the  surrounding  population.  Over 
7,000  biometric  collection  devices  were  fielded  and  used  for  functions  such 
as  detainee  management,  execution  of  high-risk  warrants,  and  targeted 
raids  against  named  insurgents.  During  the  conflict,  U.S.  forces  collected 
over  2.5  million  biometrics  records  and  placed  some  33,000  individual 
identities  onbiometrically  enabled  watch  lists  (The  Eyes  Have  It,  2012;  U.S. 
Government  Accountability  Office  [GAO],  2012). 


Biometrics  as  Doctrinal  Innovation 

The  basic  act  of  fielding  a  new  technology  by  itself  does  not  represent 
true  military  innovation.  Tools  are  not  inherently  valuable  without  a  viable 
concept  of  employment  that  describes  how  a  given  technology  will  contrib¬ 
ute  towards  achieving  an  organization’s  core  functions.  This  requires  a 
concurrent  process  of  doctrinal  innovation  that  exploits  the  potential  of  a 
new  technology  by  providing  a  theoretical  framework  and  methods  for  how 
it  will  be  used  to  achieve  military  objectives.  To  be  successful,  doctrinal 
innovation  must  occur  on  a  sufficiently  large  scale  to  overturn  old  ways  of 
doing  business,  thereby  institutionalizing  the  new  tools  and  methods  (Cote, 
1996).  This  is  no  small  task  and  sometimes  requires  a  wholesale  reconcep¬ 
tualization  of  how  an  organization  perceives  its  central  warfighting  tasks. 
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The  catalysts  for  such  change  may  come  from  a  variety  of  sources.  Some 
theories  focus  on  endogenous  factors  such  as  organizational  culture, 
civil-military  relations,  or  Service  rivalries  as  central  dynamics  in  this 
process  (Posen,  1984;  Rosen,  1991).  Other  theories  weigh  more  heavily  on 
the  influence  of  exogenous  factors  such  as  the  rise  of  unanticipated  threats 
or  emergence  of  novel  technologies  that  disrupt  the  fundamental  balance 
of  military  advantage  on  the  battlefield.  In  the  case  of  biometrics,  several 
external  factors  played  a  role  in  driving  how  these  technologies  evolved  on 
the  battlefield. 

The  U.S.  military’s  adoption  of  biometrics  emerged  within  the  context  of  a 
larger  paradigm  shift  that  moved  identity  to  the  center  of  a  new  warfighting 
paradigm.  Counterinsurgency  and  counterterrorism  operations  required  the 
U.S.  military  to  undertake  a  major  doctrinal  reorientation  focused  on  target¬ 
ing  networks  and  individual  combatants  rather  than  formations  and  weapons 
platforms.  In  his  counterinsurgency  guidance  to  multinational  forces  in  Iraq, 
Army  Gen.  David  Petraeus  directed  commanders  to  “defeat  the  network, 
not  just  the  attack”  by  focusing  intelligence  assets  on  the  nodes  and  links  of 
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the  insurgency— identifying  its  leaders,  financiers,  suppliers,  and  operators 
(Petraeus,  2008,  p.  2).  This  required  technologies  to  support  a  new  targeting 
paradigm  by  enabling  U.S.  forces  to  “identify  and  separate  the  reconcilables 
from  the  irreconcilables”  on  an  irregular  battlefield.  Biometrics  became  a 
central  technical  component  of  this  new  strategic  approach. 


Biometrics  Use  Case:  High-Value  Targeting 


An  important  aspect  of  U.S.  counterinsurgency  and  counterterrorism 
strategies  involved  identity-based  targeting  of  individual  combatants. 
Biometric  technologies  and  Biometrically  Enabled  Intelligence  (BEI) 
became  important  elements  of  the  shift  to  this  new  targeting  para¬ 
digm.  This  process  for  targeting  high-value  individuals  was  doctrinally 
formalized  within  the  find,  fix,  finish,  exploit,  analyze,  and  disseminate 
(F3EAD)  methodology.  Biometric  databases  and  watchlist  information 
played  an  important  role  in  identifying,  tracking,  and  targeting  these 
individuals.  For  example  Biometric  Identification  Analysis  Reports 
(BIAR)  provided  U.S.  forces  with  biographical  information,  encounter 
history,  and  disposition  instructions  for  persons  of  interest.  During  the 
“surge”  period  in  Iraq,  these  data  were  used  to  remove  an  average  of 
two  high-value  individuals  from  the  battlefield  each  day.  When  com¬ 
bined  with  forensic  data,  this  biometric  information  was  a  powerful  tool 
for  penetrating  cells  employing  Improvised  Explosive  Devices  against 
coalition  forces  and  matching  specific  individuals  to  these  activities. 
For  example,  from  2007  to  2008,  more  than  1,700  adversary  combat¬ 
ants  were  biometrically  linked  to  forensic  evidence  directly  associating 
them  with  the  manufacture  and  use  of  these  devices  on  the  battlefield 
(Kieffer  &  Trissell,  2010). 


As  biometric  technologies  evolved  within  this  new  warfighting  paradigm, 
DoD  directed  combatant  commanders  to  integrate  these  capabilities  into 
mission  planning  across  the  six-phase  joint  planning  model  (DoD,  2008). 
The  Army  formalized  the  doctrinal  role  for  biometrics  technologies  as 
part  of  its  concept  for  Biometrically  Enabled  Intelligence  (BEI),  or  the 
intelligence  resulting  from  the  combination  of  biometric  data  with  other 
intelligence  information  to  identify  potential  threat  actors.  The  Navy  and 
Marine  Corps  adopted  a  similar  concept  known  as  Identity  Operations 
(IdOps)  into  their  respective  Service  doctrine.  This  approach  encompasses 
the  synchronized  application  of  biometric  technologies,  forensics,  and 
identity  management  capabilities  in  support  of  maritime  and  expeditionary 
operations  (Department  of  the  Navy,  2012). 


Defense ARJ,  April  2016,  Vol.  23No.  2:178-201 


189 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


More  recently,  the  DoD  Intelligence  Community  introduced  into  joint 
doctrine  an  overarching  concept  for  Identity  Intelligence  (12),  or  the  col¬ 
lection,  analysis  exploitation  and  management  of  identity  attributes  and 
associated  technologies  and  processes  (Joint  Chiefs  of  Staff,  2013).  12 
integrates  several  distinct  technical-functional  areas  combining  BEI  with 
other  all-source  data  to  connect  individual  actors  to  other  persons,  places, 
activities,  or  materials.  This  doctrine  defines  a  specific  role  for  biometric 
technologies  across  a  range  of  mission  functions  including  raids,  checkpoint 
operations,  border  control  and  maritime  interdiction,  force  protection, 
support  to  host-nation  Rule  of  Law,  and  detailed  human  terrain  mapping. 
These  examples  all  illustrate  the  degree  to  which  biometric  technologies 
have  been  integrated  within  a  doctrinal  framework  supported  by  specific 
use  cases  and  tactical  applications. 


Biometrics  Use  Case:  Support  to  Rule  of  Law  and 
Stability  Operations 


U.S.  counterinsurgency  strategy  presented  enormous  procedural  chal¬ 
lenges  regarding  legal  adjudication  of  “unprivileged  enemy  belliger¬ 
ents”  detained  on  the  battlefield  as  well  as  monitoring  released  indi¬ 
viduals  for  recidivism.  Biometric  technologies  played  a  critical  role  in 
supporting  such  “evidence-based”  operations,  particularly  during  the 
stability  and  support  phase  when  formal  criminal  proceedings  became 
the  only  means  of  effectively  removing  insurgents  from  the  battlefield 
(Voetelink,  2013).  Biometric  and  forensic  data  provided  much  of  the 
evidentiary  basis  for  prosecution  support  packages  used  by  detainee 
review  boards  and  host-nation  criminal  proceedings  against  suspected 
insurgents.  These  packages  provided  detailed  biological  and  biograph¬ 
ical  information  linking  suspect  individuals  to  insurgent  activities. 

Counterinsurgency  strategy  also  called  for  U.S.  forces  to  help  reestab¬ 
lish  rule  of  law  and  support  local  governance.  This  included  the  transfer 
of  biometric  information  and  technologies  to  local  partners  and  train¬ 
ing  on  how  to  use  these  tools  as  part  of  legal  proceedings.  As  one 
example,  the  Afghan  government  now  maintains  its  own  biometric  da¬ 
tabase  and  uses  this  information  in  support  of  warrant-based  targeting 
and  prosecutions.  Of  recent  cases  tried  in  the  Afghan  National  Security 
Court,  there  have  been  convictions  in  a  majority  of  instances  where 
biometric  data  have  been  linked  to  forensic  evidence  presented  in  the 
case  (Pendall  &  Sieg,  2014). 
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Acquisition  and  Technology  Integration 
as  Factors  in  Military  Innovation 

The  nature  of  bureaucratic  culture  and  the  dynamics  of  the  acquisition 
process  also  play  an  important  role  in  the  process  of  military  innovation. 
As  a  general  rule,  bureaucracies  tend  towards  a  status  quo  bias;  therefore, 
they  are  not  necessarily  designed  to  accommodate  adaptation  (Samuelson 
&  Zeckhauser,  1988).  This  means  that  organizations  cannot  always  exploit 
the  full  potential  of  an  emerging  technology  even  when  there  are  clear 
advantages  over  previous  methods  (Murray,  2009).  In  the  case  of  biometrics, 
challenges  relating  to  the  acquisition  process  and  integration  of  the  new 
technologies  produced  mixed  results  in  terms  of  creating  the  conditions 
for  successful  innovation. 

In  the  initial  aftermath  of  9/11,  government  officials  immediately  recog¬ 
nized  the  need  for  improved  border  control  and  automated  systems  for 
identifying  individuals  trying  to  enter  the  country.  New  biometrics  tech¬ 
nologies  offered  one  means  of  verifying  identities  and  comparing  these 
records  against  watchlists  of  potential  threats  gathered  by  DoD  and  other 
government  agencies.  Effective  use  of  these  data  required  an  unprecedented 
effort  to  overcome  deep  institutional  barriers  between  the  Department  of 
Defense,  the  Intelligence  Community,  Homeland  Security,  and  domestic  law 
enforcement  so  that  threat  identity  information  could  be  shared  across  the 
entire  enterprise.  However,  the  U.S.  government  had  only  two  major  opera¬ 
tional  biometric  systems  on  9/11— one  at  the  Federal  Bureau  of  Investigation 
(FBI)  and  another  with  U.S.  Immigration  and  Naturalization— as  well  as  a 
handful  of  smaller  research  projects  and  pilot  studies  (National  Science  and 
Technology  Council,  2008). 

As  U.S.  forces  began  collecting  large  amounts  of  biometric  data  on  the  bat¬ 
tlefield,  a  critical  need  emerged  for  an  authoritative  database  to  process, 
store,  and  match  these  biometric  records.  This  required  an  information 
management  system  designed  for  sharing  identity  information  among 
widely  dispersed  military  forces  in  the  field,  as  well  as  with  domestic  law 
enforcement  and  the  Intelligence  Community.  Within  DoD,  this  led  to 
the  initial  prototype  design  for  what  became  the  Department  of  Defense 
Automated  Biometric  Identification  System  (DoD  ABIS),  the  military’s  cen¬ 
tralized  multimodal  biometric  data  repository.  This  system  later  included 
a  Biometrically  Enabled  Watchlist  feature  enabling  analysts  to  highlight 
person-of-interest  records,  and  provide  disposition  instructions  and  other 
relevant  information.  As  DoD  was  deploying  its  prototype  system,  the  FBI 
had  already  fielded  its  own  automated  fingerprint  database  system  known  as 
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the  Integrated  Automated  Fingerprint  Identification  System.  Concurrently, 
the  Department  of  Homeland  Security  was  conducting  an  upgrade  of  its  own 
biometric  identity  system  used  for  managing  immigration,  visa,  border  con¬ 
trol,  and  law  enforcement  requirements.  Additionally,  in  2004  the  National 
Counterterrorism  Center  was  tasked  with  managing  the  Terrorist  Identities 
Datamart  Environment,  intended  to  be  the  government’s  central  repository 
of  information  relating  to  international  terrorist  identities. 


J 


Even  as  biometric  collection  devices  proliferated  across  the  battlefields  of 
Iraq  and  Afghanistan,  DoD  struggled  to  articulate  an  overall  strategic 
vision  for  how  the  new  technologies  would  evolve  as  a  warfight¬ 
ing  capability  and  integrate  into  the  larger  national  security 
apparatus.  According  to  one  assessment,  the  DoD  biomet¬ 
rics  enterprise  lacked  “specific  and  measurable  strategic 
goals  and  objectives  for  using  biometrics”  and  a  lack  of 
common  understanding  about  the  purpose  and  bound¬ 
aries  of  the  enterprise  (Shontz,  Libicki,  Rudavsky, 

&  Bradley,  2012).  This  ambiguity  contributed  to 
discontinuities  in  the  acquisition  program  and 
criticisms  that  the  overall  DoD  biometrics  pro¬ 
gram  lacked  a  long-range  planning  horizon.  One 
specialist  working  on  biometrics  programs  at  the 
Army’s  Communications-Electronics  Research, 

Development  and  Engineering  Center  observed 
how  many  of  the  Quick-Reaction  Capabilities 
fielded  during  the  conflicts  were  only  used  for  a 
year  or  two,  then  not  sustained  due  to  shrinking  bud¬ 
gets  or  changing  operational  priorities  (Jontz,  2015). 

In  the  case  of  biometrics,  the  focus  on  rapidly  moving 
collection  devices  out  to  units  also  meant  that  some  new 
capabilities  were  fielded  without  adhering  to  DoD  standards, 
performance  measures,  and  operational  testing  and  evaluation 
requirements  (Shontz  et  al.,  2012). 


The  rapid  fielding  process  also  had  implications  regarding  preparing  the 
force  for  integration  of  the  new  technologies.  Because  these  technologies 
were  a  relatively  untested  capability,  the  military  had  not  yet  developed  the 
human  capital  needed  to  fully  exploit  their  potential.  Initially,  a  relatively 
small  number  of  trained  users  and  leaders  were  familiar  with  the  systems. 
For  example,  the  GAO  found  that  DoD  did  not  sufficiently  instruct  unit 
commanders  on  effective  use  of  biometrics,  and  noted  that  many  military 
leaders  were  unaware  of  how  the  technology  contributed  to  identifying 
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enemy  combatants  (GAO,  2012).  This  led  to  confusion  over  how  and  when 
to  incorporate  biometrics  capabilities  into  mission  planning  and  how  to 
best  employ  the  systems  in  the  field.  A  separate  study  attributed  some  of 
these  shortfalls  to  delays  in  establishing  biometrics  as  a  formal  Program 
of  Record  that  would  have  formalized  the  process  of  establishing  common 
training  standards  (Shontz  et  al.,  2012). 


While  some  units  such  as  Special  Operations  forces  clearly  leveraged  the 
new  technology  to  great  effect,  its  operational  integration  across  the 
force  was  uneven.  Inconsistent  training  meant  that  individual 
units  applied  significant  discretion  in  terms  of  what  biometric 
data  were  gathered  and  the  methods  of  collection.  These 
training  shortfalls  affected  the  quality  of  biometric  data 
collection,  and  in  some  cases  resulted  in  the  loss  of 
information  gathered  from  the  field  and  delays  in 
transmission  into  the  centralized,  authoritative 
database  (GAO,  2012).  In  hindsight,  rapid  fielding 
was  the  correct  decision  from  the  perspective 
of  supporting  soldiers  in  the  field  with  avail¬ 
able  technology;  however,  it  was  not  without 
consequences.  The  process  was  likely  a  factor 
contributing  to  challenges  with  interoperability 
and  training  that  ultimately  limited  the  oper¬ 
ational  impact  of  a  promising  new  technology. 

Other  problems  encountered  during  the  early 
deployment  of  biometrics  were  not  specifically 
related  to  the  technology  itself,  but  rather  reflected 
bureaucratic  challenges  involved  in  the  acquisition 
process.  Discussions  with  DoD’s  Biometrics  program 
manager  suggested  that  the  Executive  Agent  was  not  suffi¬ 
ciently  empowered  to  provide  effective  oversight  and  strategic 
guidance  across  the  enterprise  as  the  technology  evolved  (Vann- 
Olejasz,  personal  communication,  2014-2015).  This  contributed  to 
challenges  promulgating  and  enforcing  standards  of  interoperability  as 
various  components  pursued  independent  development  programs  (GAO, 
2012;  Shontz  et  al.,  2012).  For  example,  by  2011  the  Army  had  still  not  fully 
adopted  common  biometric  standards  for  its  primary  handheld  collection 
device,  the  HIIDE,  being  used  in  Iraq  and  Afghanistan.  This  left  the  system 
unable  to  automatically  transmit  biometric  data  to  other  federal  agencies. 
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According  to  the  GAO,  since  the  device  was  developed  in  response  to  an 
urgent  mission  requirement,  it  was  not  required  to  adhere  to  DoD’s  infor¬ 
mation  technology  standards. 

Other  difficulties  emerged  related  to  coordination  among  a  diverse  range 
of  users,  often  with  differing  technology  requirements  and  protocols  for 
handling  biometric  information.  According  to  the  GAO,  system  capacities 
developed  for  different  mission  needs  affected  agencies’  ability  to  process 
one  another’s  queries  for  biometric  information.  This  complicated  the  pro¬ 
cess  of  developing  and  approving  interagency  biometric  sharing  agreements 
between  DoD  and  the  FBI.  Similar  problems  were  encountered  establishing 
direct  connectivity  between  DoD  and  Department  of  Homeland  Security 
(DHS)  biometric  databases  (GAO,  2011).  Even  within  DoD,  various  com¬ 
ponents  were  not  always  able  to  seamlessly  share  biometric  information 
using  a  commonly  understood  process  and  methodology.  This  issue  included 
challenges  of  passing  and  comparing  information  stored  on  domains  of  dif¬ 
ferent  classification.  These  examples  support  Williamson  Murray’s  (2009) 
contention  that  technology  implementation  is  an  equally  important  aspect 
of  military  innovation  as  the  sophistication  of  the  technology  itself. 


Innovation  Lessons  Learned  from 
Defense  Biometrics 

As  a  recent  example  of  military  innovation,  biometrics  offers  a  useful 
case  study  for  understanding  how  a  new  and  relatively  untested  technology 
was  integrated  into  operational  use  during  wartime.  At  the  start  of  com¬ 
bat  operations  in  Iraq  and  Afghanistan,  the  U.S.  military  had  virtually  no 
experience  or  operational  concepts  for  employing  biometrics.  However,  by 
the  end  of  the  decade  the  devices  had  become  a  commonplace  tool  on  the 
battlefield  and  an  important  enabling  technology  of  counterinsurgency  and 
counterterrorism  operations. 

Several  factors  contributed  to  this  outcome.  First,  within  the  context 
of  the  unique  tactical  challenges  encountered  by  U.S.  forces  in  Iraq  and 
Afghanistan,  biometric  technologies  had  a  number  of  specific  and  highly  rel¬ 
evant  use  cases.  Second,  the  technology  was  firmly  grounded  in  a  doctrinal 
framework  and  overarching  warfighting  strategy  that  clearly  articulated 
how  the  technology  could  be  used  to  improve  the  effectiveness  of  U.S.  forces 
on  the  battlefield.  Third,  during  the  initial  developmental  stage,  multiple 
constituencies  actively  pushed  for  the  integration  of  biometrics  technologies 
for  a  wide  variety  of  applications.  Finally,  DoD  and  other  users  benefitted 
from  a  rapidly  expanding  commercial  marketplace  that  was  able  to  deliver 
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cutting-edge  technologies,  readily  adaptable  to  military  use.  The  combina¬ 
tion  of  these  factors  played  a  significant  role  as  catalysts  for  innovation  and 
facilitated  the  relatively  successful  integration  of  a  new  military  technology. 

However,  despite  these  significant  achievements,  biometrics  was  not  a 
flawless  example  of  military  innovation.  Some  notable  shortfalls  related  to 
challenges  associated  with  the  rapid  fielding  process.  For  example,  the 
urgent  demand  to  move  collection  devices  out  to  units  meant  that  some  new 
technologies  were  deployed  without  adhering  to  formal  performance  mea¬ 
sures  and  standards  for  interoperability.  This  contributed  to  difficulties  in 
moving  and  sharing  biometric  information  among  interagency  partners. 
Additionally,  as  the  new  tools  were  placed  into  units,  initially  a  relatively 
limited  number  of  users  and  leaders  possessed  sufficient  knowledge  and 
experience  to  fully  exploit  the  potential  of  the  new  technology.  These  chal¬ 
lenges  were  certainly  not  limited  to  DoD.  Indeed,  one  group  of  experts 
recently  noted  that  even  as  biometrics  technologies  rapidly  evolved  over  the 
last  decade,  the  legal,  political,  and  resource  framework  for  how  to  imple¬ 
ment  these  tools  has  lagged  behind  the  technological  advances 
(Aughenbaugh,  2015). 


In  terms  of  rapidly  developing  and  fielding  a  new 
technology,  the  record  of  defense  biometrics  should  be 
considered  a  tactical  success. 


In  terms  of  rapidly  developing  and  fielding  a  new  technology,  the  record 
of  defense  biometrics  should  be  considered  a  tactical  success.  During  the 
course  of  the  conflicts  in  Iraq  and  Afghanistan,  U.S.  forces  generally  made 
effective  use  of  an  emerging  capability  that  directly  enabled  new  forms  of 
identity-based  operations  in  response  to  unique  demands  of  waging  irregu¬ 
lar  warfare.  However,  the  rapid  fielding  process  did  reveal  shortcomings  in 
how  DoD  manages  military  innovation  at  the  bureaucratic  level.  These  chal¬ 
lenges  are  undoubtedly  not  unique  to  biometrics  and  are  certainly  worthy 
of  future  study  to  better  understand  how  DoD  can  improve  process  models 
for  wartime  innovation.  As  one  recent  study  of  military  innovation  noted, 
militaries  exist  for  war,  but  they  more  often  innovate  during  peacetime 
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(Hill,  2015).  Therefore,  strategies  for  innovation  must  be  adaptable  to  both 
environments  and  able  to  survive  the  transition  from  one  condition  to  the 
next.  In  the  end,  this  may  be  one  of  the  key  lessons  learned  from  the  example 
of  bio  metrics. 


Challenges  for  the  Future 

The  lessons  drawn  from  the  initial  experience  of  fielding  biometrics  will 
be  particularly  important  as  the  technology  enters  its  second  generation— an 
evolution  that  will  most  likely  progress  along  a  very  different  developmental 
path  than  the  initial  phase.  In  this  respect,  biometrics  may  offer  an  exam¬ 
ple  of  the  changing  model  for  development  and  acquisition  of  cutting-edge 
defense  technologies.  During  the  Cold  War  era,  DoD  developed  many  of  its 
most  important  capabilities  within  a  closed  system  of  innovation  dominated 
by  the  defense-industrial  complex.  Most  of  these  technologies  were  created 
under  the  purview  of  government-sponsored  research  and  development 
programs,  built  in  collaboration  with  a  relatively  small  circle  of  defense 
contractors.  An  emerging  model  of  military  innovation  may  increasingly 
involve  awider  range  of  commercial  providers  developing  new  technologies 
not  explicitly  designed  for  defense  applications,  but  later  adapted  to  military 
purposes.  The  field  of  biometrics  reflects  the  dynamics  of  this  transition. 

The  attacks  of  9/11  and  subsequent  conflicts  in  Iraq  and  Afghanistan  were 
important  initial  catalysts  driving  the  first  biometrics  revolution.  Between 
2007  and  2015,  DoD  drove  a  sizable  portion  of  new  investments  in  the  field 
with  an  estimated  $3.5  billion  in  program  spending  (GAO,  2011).  These 
requirements  substantially  defined  many  of  the  initial  prototype  technol¬ 
ogies  that  fueled  industry  growth  rates  in  excess  of  28  percent  between 
2005  and  2010  (Gelb  &  Clark,  2013).  However,  even  during  this  period  of 
rapid  expansion,  already  underway  was  a  gradual  transition  of  the  cus¬ 
tomer  base— away  from  government  and  military  requirements.  As  the 
sector  matured,  it  shifted  towards  new  applications  in  health  care,  retail 
services,  banking,  and  consumer  digital  devices  (Biometrics  Gets  Down  to 
Business,  2006).  This  trend  is  only  expected  to  accelerate  as  DoD  represents 
an  increasingly  smaller  fraction  of  this  rapidly  expanding  marketplace. 

One  recent  industry  report  placed  the  value  of  the  current  global  biometrics 
market  at  $7  billion  annually,  projected  to  reach  $44  billion  per  year  by  2021. 
However,  the  key  growth  areas  for  the  industry  will  likely  come  from  sectors 
other  than  military  and  defense.  Furthermore,  the  United  States  will  not 
be  the  primary  driver  of  this  growth  with  countries  such  as  India,  Mexico, 
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Russia,  and  China  expected  to  create  much  of  the  future  demand  for  bio¬ 
metrics  technologies  (National  Security  and  Market  Watch,  2015;  King, 
2014).  What  this  means  in  practical  terms  is  that  DoD  will  increasingly  need 
to  look  beyond  the  traditional  jurisdiction  of  government-sponsored 
research  and  development  programs  to  access  cutting-edge  technologies  in 
the  field.  This  will  be  particularly  true  across  the  range  of  research  areas 
likely  to  be  critical  for  the  next  generation  biometrics  capabilities— areas 
such  as  remote  sensing,  data  science  and  artificial  intelligence,  information 
management,  and  communications.  All  of  these  factors  suggest  that  future 
military  innovation  will  depend  largely  on  DoD’s  ability  to  identify  and 
effectively  assimilate  commercial  technologies  from  the  nondefense  sector. 
The  lessons  from  biometrics  suggest  a  few  of  the  potential  challenges. 


One  recent  industry  report  placed  the  value  of  the 
current  global  biometrics  market  at  $7  billion  annually, 
projected  to  reach  $44  billion  per  year  by  2021. 


Biometrics,  in  particular,  is  a  technology  where  the  benefits  derive  from 
network  effects,  meaning  that  its  utility  is  directly  related  to  the  number 
of  users  able  to  input  data,  conduct  searches,  and  discover  associations 
within  a  commonly  accessible  database.  This  makes  interoperability 
central  to  the  value  proposition  of  the  technology.  As  the  last  decade  of 
counterinsurgency  and  counterterrorism  operations  demonstrated,  U.S. 
national  security  strategy  increasingly  requires  a  ’’whole  of  government” 
approach  based  on  seamless  information  sharing  between  the  military. 
Intelligence  Community,  State  Department,  DHS,  and  law  enforcement. 
Furthermore,  transnational  concerns  about  terrorism,  organized  crime, 
and  mass  migrations  will  require  expanded  collaboration  and  greater  infor¬ 
mation  sharing  across  borders  and  between  governments  in  the  future.  The 
issues  of  interoperability  and  technology  integration  will  be  increasingly 
critical  aspects  of  innovation  as  governments  adopt  strategies  based  on 
data-intensive  decision  making. 

Given  the  rate  of  change  in  the  commercial  sector,  DoD  will  be  challenged 
to  keep  pace  with  new  developments,  continuous  upgrades  to  existing  sys¬ 
tems,  and  the  rapid  evolution  of  new  applications  for  existing  technologies. 
Furthermore,  some  of  the  initiatives  intended  to  spur  innovation  such 
as  greater  service  autonomy  in  acquisition,  increased  prototyping,  and 


Defense ARJ,  April  2016,  Vol.  23No.  2:178-201 


197 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


accelerated  fielding  may  even  exacerbate  existing  challenges  regarding 
interoperability,  data  sharing,  and  integration.  This  also  raises  concerns 
about  whether  doctrinal  development,  concepts  of  employment,  and  force 
training  can  keep  up  with  the  pace  of  technological  advances.  These  issues 
highlight  the  fact  that  identifying  and  acquiring  cutting-edge  technology 
is  only  one  aspect  of  successful  military  innovation. 
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The  federal  government  persistently  fails  to  make  service  contracts  a 
managed  outcome.  Consequently  the  three  objectives  of  public  procurement 
(transparency  value  for  money  and  meeting  requirements)  are  jeopardized. 
This  research  identifies  the  culprits  as  methodologies  that  are  incompatible 
with  the  characteristics  of  services.  These  methodologies  involve  best-value 
source  selection  and  contractor  performance -information  collection  and 
evaluation.  A  new  method  of  best-value  proposal  evaluation  is  offered  that 
enables  the  buying  agency  to  validly  measure  service  quality  then  to  trade 
off  levels  of  service  quality  with  price,  resulting  in  a  Quality-Infused  Price 
(QIP)®.  The  concept  is  tested  on  a  task  order  competition  using  a  case  study 
methodology.  Findings  suggest  that  service  quality  can  be  monetized  and 
that  the  application  of  a  QIP®  methodology  can  result  in  a  superior  sourcing 
decision.  Additionally  fewer  and  higher  quality  proposals  will  be  received. 
Based  on  the  findings,  conclusions  are  drawn  and  suggestions  for  future 
research  are  offered. 
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Over  the  past  several  decades,  the  United  States  transitioned  from 
a  goods-based  to  a  services-based  economy  (McCullough,  2012;  Powell 
&  Snellman,  2004).  As  of  2013,  services  accounted  for  78  percent  of  the 
country’s  gross  domestic  product  and  employed  82  percent  of  the  country’s 
workforce  (U.S.  International  Trade  Commission,  2015).  At  68  percent  of 
total  contract  spending  in  2014,  federal  spending  on  services  is  substantial 
(Schwartz,  Ginsberg,  &  Sargent,  2015).  The  Department  of  Defense  (DoD) 
obligated  45  percent  of  its  contract  spend  on  services— an  equal  proportion 
as  that  spent  on  goods  (Schwartz  et  al.,  2015).  While  the  DoD  increasingly 
relies  on  defense  contractors  for  services,  it  lacks  the  key  elements  at  the 
strategic  and  tactical  levels  to  make  service  contracts  a  managed  outcome 
(U.S.  Government  Accountability  Office  [GAO],  2007a).  Improving  the 
tradecraft  in  services  contracting  has  been  a  federal  focus  for  some  time 
(GAO,  2006;  GAO  2007b;  GAO,  2009b;  Kendall,  2015;  GAO,  2001a,  2001b). 

Service  has  been  defined  as  “the  application  of  specialized  competences 
(knowledge  &  skills)  through  deeds,  processes,  &  performances  for  the 
benefits  of  another  entity  or  the  entity  itself”  (Vargo  &  Lusch,  2004,  p.  2). 
The  Federal  Acquisition  Regulation  (FAR)  defines  a  service  contract  as  a 
“contract  that  directly  engages  the  time  and  effort  of  a  contractor  whose 
primary  purpose  is  to  perform  an  identifiable  task  rather  than  to  furnish  an 
end  item  of  supply”  (subpart  37.101).  Services  are  characterized  as  complex, 
heterogeneous,  intangible  (Apte,  Ferrer,  Lewis,  &  Rendon,  2006),  perishable, 
and  inseparable  (Ellram,  Tate,  &  Billington,  2007).  First,  the  intangible 
nature  of  services  renders  specifications  and  customer  expectations  to 
be  imprecise  (Ellram  et  al.,  2007).  Second,  services  are,  by  nature,  het-  | 
erogeneous  (Hawkins,  Muir,  &  Hildebrandt,  2011).  This  is  especially 
true  of  services  with  a  high  labor  content,  as  performance  will  vary 
between  providers  and  will  likely  differ  between  customers  and  with 
time  required  to  deliver  services  (Parasuraman,  Zeithaml,  &  Berry,  s.  M 
1985).  Like  providers,  customers  also  lack  a  homogenous  definition  ' 
of  service  quality  for  many  specified  services  (Hawkins,  Berkowitz, 

Muir,  &  Gravier,  2015).  Because  of  this,  and  since  consistency  in 
levels  of  performance  from  service  personnel  is  difficult  to  attain, 
the  level  of  quality  that  a  service  provider  expects  to  deliver  may  jjL 
vary  greatly  from  the  level  of  quality  that  the  customer  expects  to  "K. 

receive  (Parasuraman  et  al.,  1985).  Third,  services  are  frequently 
perishable;  unlike  goods,  services  cannot  be  held  or  stocked  in 
inventory.  Whereas  inventory  policies  for  goods  allow  firms  to 
buffer  variability  in  future  demand  with  safety  stock,  service 
providers  must  change  service  capacity  to  meet  demand  flue-  f 
tuations  (Ellram  et  al.,  2007).  The  perishability  of  services 
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also  presents  challenges  for  inspection;  service  outcomes 
for  many  services  can  be  inspected  or  evaluated  only  at  the 
time  of  service  performance  (Hawkins  et  al,  2011).  Given  these 
perplexing  challenges,  how  does  the  government  validly  lever¬ 
age— not  eliminate— the  use  of  subjective  service  quality  in  both 
selecting  contractors  (i.e.,  reduce  the  risk  of  adverse  selection)  and 
motivating  their  performance  (i.e.,  reduce  the  risk  of  moral  hazard)? 

To  explore  this  question,  this  research  supposes  two  axioms  sur¬ 
rounding  the  objectives  of  procuring  activities.  First,  the  three  primary 
objectives  of  public  procurement  are:  transparency  (Gilbert,  Schapper,  & 
Veiga-Malta,  2009),  value  for  money  (Gilbert  et  al.,  2009),  and  meeting 
agency  requirements.  Second,  the  ability  to  procure  services  effec¬ 
tively  and  efficiently  is  desirable  and  in  the  public’s  best  interest 
(Gilbert  et  al.,  2009). 

Agencies  constantly  trade  these  objectives  based  on  risk  that  consid¬ 
ers  how  best  to  meet  agency  requirements,  gain  value  for  money,  and 
maintain  a  transparent  process.  These  three  objectives  do  not  operate 
in  a  vacuum;  rather,  they  are  interoperable.  If  the  agency  can  clearly  and 
efficiently  articulate  requirements  and  evaluation  methodologies,  that 
clarity  should  increase  transparency  and,  thereby,  reduce  the  risk  of  bid 
protest  (i.e.,  delayed  meeting  agency  requirements).  That  clarity  should  also 
enable  offerors  to  propose  best  in  accordance  with  the  agencies’  needs  (i.e., 
yield  more  value  for  money).  Agencies  must  strategically  assess  the  three 
objectives  when  they  determine  their  source  selection  methodology  along 
the  best -value  spectrum,  ranging  from  lowest  price  technically  acceptable 
(LPTA)  to  full  trade-off  of  price  and  nonprice  factors. 


The  ability  to  define,  evaluate,  select,  award,  and  then  manage  service 
contracts  is  a  problem  that  has  garnered  significant  attention  in  the  past 
decade,  yet  the  many  identified  problems  have  not  been  resolved.  Further, 
related  problems  in  these  areas  and  industry  feedback  have  led  to  a  call  to 
“monetize”  trade-offs  to  allow  industry  to  understand  the  relative  impor¬ 
tance  of  evaluation  factors  in  a  manner  that  equates  quality/performance 
to  dollars  of  value  (Kendall,  2015).  The  purpose  of  this  research,  therefore, 
is  to  provide  a  new  approach  that  bridges  the  best-value  continuum  divide, 
optimizes  the  three  primary  objectives  of  public  procurement,  and  delivers 
best  value  to  the  public  sector  by  accounting  for —rather  than  ignoring— the 
inherently  subjective  valuation  of  services. 
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Source  Selection 

FAR  15.304  identifies  that  price/cost,  quality  of  the  product  or  service, 
and  past  performance  must  be  evaluated  when  contracting  by  negotiation. 
These  criteria  are  considered  across  the  best-value  continuum  spanning 
from  LPTA  to  full  trade-off  in  which  the  noncost  factors  maybe  significantly 
more  important  than  cost/price.  Agencies  must  know  their  requirements 
well  enough  to  establish  the  best  evaluation  approach  for  source  selection 
across  this  spectrum  considering  the  three  aforementioned  objectives. 
LPTA  generally  compares  to  full  trade-off  as  shown  in  Appendix  A. 

Federal  acquisition  and  industry  professionals  have  noted  the  following 
issues  with  LPTA  versus  full  trade-off  (Watson,  2015). 

•  Full  trade-off  evaluations  may  be  too  complex  if  workforce 
experience  is  low. 

•  Less  procurement  administrative  lead  time  (PALT)  is  a  driving 
factor  in  using  LPTA. 

•  Evaluation  criteria  need  to  be  better  defined  to  industry. 
Industry  needs  to  know  the  relative  weight  of  cost/price  to 
trade  factors. 

•  Industry  needs  to  know  the  level  of  performance  to  offer. 

•  Industry  needs  help  understanding  the  competitive  effects  of 
a  higher  performance  offer. 

•  There  is  a  desire  to  avoid  protests  (Hawkins,  Gravier,  & 
Yoder,  in  press).  Agencies  must  create  meaningful  evaluation 
discriminators. 

•  Industry  can’t  determine  the  buying  agencies’  priorities. 

•  LPTA  is  perceived  as  “low  cost/low  quality.” 

•  Cost  risk  does  not  equate  to  proposal  risk. 

The  acquisition  team’s  challenge  is  to  find  the  optimal  point  within  the 
best -value  continuum  to  deal  with  these  issues.  The  means  by  which  federal 
agencies  deal  with  the  primary  indication  of  quality  and  value  for  money- 
past  performance— must  first  be  explored. 
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Contractor  Performance  Rating  System 

The  GAO  asserted  that  contractor  performance  reports  should  be 
timely,  accurate,  and  complete  to  allow  federal  procurement  officials  to 
make  informed  source  selection  decisions  in  the  future.  Despite  persistent 
attention  from  the  Office  of  Federal  Procurement  Policy  (OFPP)  and  the 
GAO,  Contractor  Performance  Assessment  Reports  (CPAR)  continue  to  be 
plagued  by  a  lack  of  reporting,  untimeliness,  incompleteness,  and  inaccu¬ 
racies  (GAO,  2009a,  2014b;  Gordon,  2011).  Agencies  reported  that  workforce 
shortages,  work  priorities,  time  constraints,  and  difficulty  obtaining  timely 
feedback  from  other  parts  of  the  acquisition  workforce  are  affecting  report¬ 
ing  compliance  (GAO,  2013,  2014b). 


U.S.  Federal  Government  agencies  use  the  Contractor 
Performance  Assessment  Reports  System  (CPARS)  to 
collect  contractor  performance  information  and  use 
the  Past  Performance  Information  Retrieval  System  to 
access  it. 

U.S.  Federal  Government  agencies  use  the  Contractor  Performance 
Assessment  Reports  System  (CPARS)  to  collect  contractor  performance 
information  and  use  the  Past  Performance  Information  Retrieval  System  to 
access  it.  Additional  information  that  helps  buyers  reduce  the  risk  of  adverse 
selection  is  available  from  the  Federal  Awardee  Performance  and  Integrity 
Information  System.  The  CPARS  scores  contractors  using  a  rating  system 
of  criteria  including  quality,  schedule,  cost  control,  management,  and  small 
business  utilization  (GAO,  2014b).  To  address  the  weaknesses  of  the  CPARS, 
assessing  “reputation  attributes”  (Blott,  Boardman,  Caday,  Elliott,  Griffin, 
Mastronardi,  &  Quinn,  2015)  has  been  suggested  to  more  closely  align  to 
commercial,  “crowd-sourced”  supplier  performance  evaluations.  Such 
evaluations  are  updated  in  real  time,  known  as  “point-of-service,”  as  seen 
with  online  platforms  such  as  Amazon,  Yelp,  Foursquare,  etc.  (Whetsell, 
2015).  Point-of-service  platforms  allow  the  customer  to  essentially  “score” 
the  vendors  on  their  subjective  experience,  based  on  objective  realities,  in  a 
timely  manner  that  can  lead  to  increased  accuracy  of  reporting  (Whetsell, 
2015).  Currently,  CPARS  reporting  occurs  annually  with  a  60-day  contrac¬ 
tor  review  window.  Reports  are  not  required  until  120  days  following  the 
first  365 -day  period  of  performance.  This  means  that  it  can  take  up  to  485 
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days  to  officially  capture  the  service  quality  delivered  for  an  annual  service 
(CPARS,  2015).  Crowd-sourced,  point-of-service  reporting  leads  to  a  more 
holistic  view  of  the  contractor’s  performance  in  near  real-time. 

For  instance,  some  customers  are  satisfied  with  a  small  proportion  of  late 
deliveries.  Yet,  others  are  upset  with  a  contractor’s  inability  to  perform  to 
all  of  the  terms  of  the  original  contract.  Both  customers  are  receiving  the 
same  objective  performance  in  the  late  delivery,  but  they  may  reach  two 
different  scores  when  rating  the  contractor.  Many  would  view  this  level  of 
subjectivity  as  a  flaw  in  the  rating  system;  however,  the  subjective  reputation 
scoring  embeds  and  assesses  the  contractor’s  ability  to  manage  across  all 
customers  and  demonstrates  a  truer  measure  of  the  contractor’s  customer 
management  abilities.  How  well  a  contractor  manages  relationships  across 
its  market  share  of  customers  becomes  apparent  in  this  type  of  performance 
evaluation.  This  may  best  demonstrate  the  risk  of  partnering  with  the  con¬ 
tractor  on  future  service  needs  considering  their  ability  to  balance  their 
customer  relationship  priorities.  Such  scoring  methodologies  account  for 
the  aforementioned  characteristics  of  services— complexity,  heterogeneity, 
and  intangibility.  This  methodology  may  increase  the  chances  of  obtaining 
value  for  money  in  an  efficient  and  effective  manner  while  meeting  the 
agency’s  requirements. 


Quality-Infused  Price  (QIP)® 

Tying  the  best-value  source  selection  method  and  contractor  past  per¬ 
formance  rating  together  “monetizes  the  trade-off”  (Kendall,  2015).  The 
source  selection  method  used  and  the  performance  assessment  method 
used  must  enable  industry  to  understand  the  competitive  effects  of  higher 
performance  offers  and  to  discern  the  level  of  performance  to  propose 
in  response  to  a  solicitation  (Watson  2015).  Benefits  of  monetizing  the 
trade-off,  aside  from  the  expected  better  value  offer,  include  faster  PALTs 
and  reduced  protest  risk.  Recently,  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics,  or  USD(AT&L),  pointed  out  that, 
although  the  number  of  DoD  protests  have  increased  in  recent  years  (from 
2001  to  2013),  the  sustainability  rate  of  those  protests  has  dramatically 
declined.  The  USD(AT&L)  concluded  that  the  Better  Buying  Power  initia¬ 
tive  to  define  value  better  in  “best  value”  maybe  a  significant  contributor  to 
that  success  (Defense  Science  Board  Task  Force,  2011).  Monetizing  trade¬ 
offs  more  clearly  defines  this  value  in  terms  of  dollars.  The  proposed  QIP® 
method  aligns  with  and  expounds  upon  the  Better  Buying  Power  initiatives. 
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The  proposed  QIP®  concept  addresses  all  three  of  the  public  procurement 
objectives— transparency,  value  for  money,  and  meeting  agency  require¬ 
ments.  While  the  USD(AT&L)’s  direction  is  heavily  concerned  with  defining 
how  much  more  buying  agencies  would  pay  for  performance  of  a  system  or 
“thing,”  it  falls  short  in  defining  monetized  trade-offs  for  something  as  com¬ 
plex,  heterogeneous,  and  intangible  as  a  service.  Evaluating  services  requires 
a  midpoint  between  LPTA  and  full  trade-off.  Such  a  methodology  should  seek 
to  give  the  combined  benefit  of  faster  evaluation  processes  (meeting  agency 
requirements),  more  clear  criteria  to  aid  industry  in  deciding  howto  position 
the  quality  of  their  offer  versus  the  costs  of  their  offer  (value  for  money),  and 
an  understanding  of  the  agency’s  award  decision  (transparency). 


Further,  the  QIP®  methodology  should  not  end  with  the  award 
decision.  Components  of  the  methodology  should  be  used 
with  assessing  contractor  past  performance  in  a 
way  that  becomes  a  program  of  record,  or 
“score,”  for  each  firm.  Consider  an  indi¬ 
vidual’s  credit  score.  Credit  scores  (e.g., 

FICO  scores)  quantitatively  encapsu 
late  past  financial,  contractual,  and 
behavioral  performance  to  indi 
cate  the  risk  of  loss  of  lending 
to  an  individual.  A  similar 
model  can  be  used  by 
federal  agencies  to 
determine  the  qual¬ 
ity  risk,  management 
risk,  cost  risk,  and 
“other”  risks  related 
to  trusting  a  particular 
firm— the  firm’s  “reputa¬ 
tion”  currency. 

To  find  the  previously  discussed 
midpoint  on  the  best -value  con¬ 
tinuum,  we  propose  the  use  of  a 
composite  Quality  Adjustment 
Factor  (cQAF)  in  developing  a  QIP 
(i.e.,  an  evaluated  price  adjusted  for  ser 
vice  quality)  (Finkenstadt,  2015).  Such  a  measure 
provides  for  faster  PALT,  more  clear  criteria  for  award, 
monetizes  the  trade-off  for  industry,  and  creates  a  past 
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performance  standard  that  more  closely  aligns  with  the  shift  in  commercial 
performance  management.  This  system  can  open  the  door  to  new  ways  of 
conducting  source  selections  while  adding  the  post- award  benefits  found  in 
incentive  contracting  to  all  forms  of  service  procurement. 


This  system  can  open  the  door  to  new  ways  of 
conducting  source  selections  while  adding  the 
postaward  benefits  found  in  incentive  contracting  to 
all  forms  of  service  procurement. 


The  cQAF  described  previously  is  a  factor  that  may  be  greater  to,  equal  to, 
or  less  than  1.  It  is  derived  from  subjective  service  quality  measures.  The 
cQAF  is  used  to  assign  a  relative  level  of  quality  to  the  proposed  price,  con¬ 
sidering  factors  determined  to  be  germane  to  service  value  to  the  agency 
(Finkenstadt,  2015). 

Once  an  offeror’s  prices  are  determined  to  be  fair  and  reasonable,  the  agency 
applies  the  cQAF  to  the  prices.  Followingthe  intent  of  FAR  15.304(c)(2),  the 
agency  would  evaluate  the  quality  of  services  being  proposed  by  each  offeror. 
To  establish  a  value  rating  commensurate  with  the  quality  of  the  services 
being  offered,  the  agency  may  use  one  or  both  parts  of  this  two  -part  method¬ 
ology,  as  shown  in  Table  1.  The  first  part  assesses  relevant  past  performance, 
and  consists  of  developing  a  composite  Service  Value  Index  (cSVI)  using 
survey  data  from  the  offeror’s  previous  customers  (Finkenstadt,  2015). 
Contracts  that  leverage  post-award  incentives  such  as  award  fees  and 
incentive  fees  can  be  considered  in  the  establishment  of  this  cSVI,  either  by 
having  such  ratings  impact  the  score  or  subjectively  as  raters  consider  such 
factors  in  determining  their  level  of  satisfaction.  This  element  remains  to  be 
codified  and  could  depend  on  the  type  of  service.  The  second  part  assesses 
the  quality  of  the  offeror’s  proposal  considering  relevant  service  quality 
indicators  particular  to  the  requirement  such  as  personnel  qualifications, 
technical  process  excellence,  and  management  capability.  The  second 
part  results  in  developing  a  composite  Proposal  Quality  Rating  (cPQR) 
(Finkenstadt,  2015). 
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personnel  capacity  management  proposed? 

How  satisfied  are  you  with  the  quality  of  4  0.95 

demand  planning  processes  proposed? 

How  satisfied  are  you  with  the  quality  of  2  1.10 

SUBCATEGORY  continuous  process  improvement  proposed? 


TABLE  1.  CQAF  CATEGORY  RATING  TOOL,  CONTINUED 
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To  establish  the  final  cQAF,  cSVI,  and  cPQR,  first  a  scaled  rating  system 
that  converts  subjective  service  quality  into  objective  factors  is  needed. 
This  is  the  moment  in  which  the  trade-off  is  monetized.  As  such,  these  fac¬ 
tors  should  not  be  established  arbitrarily.  Each  type  of  service  should  be 
investigated  using  market  research  to  determine  the  appropriate  amount  of 
“value  for  money”  that  each  level  of  service  quality  represents  to  a  majority 
of  customers.  This  value-for-money  scale  may  be  created  through  market 
research  into  leading  performance  indicators  in  a  particular  type  of  service. 
A  simplified  five-point  Likert  scale  is  offered  in  Table  2.  Note  that  the  scaling 
creates  “golf-like”  reverse  indices  that  increase  with  negative  ratings  and 
decrease  with  positive  ratings  (Finkenstadt,  2015). 


TABLE  2. 

CQAF  RATING  SCALE  (NOTIONAL)  [ 

Adjective  Rating 

Numerical  Rating 

cQAF 

Strongly  Disagree 

1 

11 

Somewhat  Disagree 

2 

1.05 

Neither  Agree  Nor 
Disagree 

3 

1.0 

Somewhat  Agree 

4 

0.95 

Strongly  Agree 

5 

0.90 

Not  Rated 

None 

Not  Included 

The  cSVI  is  the  factor  that  would  become  the  crowd-sourced  reputation 
score  (i.e.,  the  “numerical  rating”  listed  in  Table  2).  A  cSVI  survey  should 
be  developed  using  an  established  scale  with  valid  psychometric  properties. 
The  service  quality  scale  included  in  Table  2  was  recently  developed  for  a 
business-to-business  context  (Hawkins  et  al.,  2015),  but  may  need  further 
refinement  by  type  of  service  (i.e.,  design-engineering  services,  testing 
services,  facility  management  services,  etc.).  These  assessments  would  be 
subjective  in  nature  and  are  intended  to  systematically  capture  the  quality 
of  a  particular  firm  operating  within  the  type  of  service  as  assessed  by  the 
most  recent  and  relevant  customers.  This  assessment  would  be  solely  at 
the  agency’s  discretion  in  determining  best  value  for  each  requirement. 
This  part  of  the  cQAF  could  replace  the  fallible  (Blott  et  al.,  2015)  CPARS. 
The  cSVI  could  be  used  for  near  real-time  ratings  that,  even  if  constrained 
by  the  current  vendor  60-day  review  window,  would  reduce  final  service 
performance  reporting  by  up  to  88  percent  when  compared  to  the  maximum 
CPARS  annual  reporting  window  of 485  days  (CPARS,  2015). 
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The  cPQR  is  unique  to  each  acquisition  and  may  or  may  not  be  used  in 
addition  to  the  cSVI  to  establish  the  cQAF.  It  should  be  established  using 
questions  for  the  technical/quality  evaluation  team  members  to  consider  in 
scoring  each  proposal.  This  would  be  similar  to  the  areas  that  are  considered 
significant  technical  subfactors  within  a  proposal.  The  agency  would  then 
derive  the  final  cQAF  to  be  used  to  establish  the  QIP®  by  combining  the  cSVI 
and  cPQR  factors  using  an  agency-determined  weight  of  importance  per  fac¬ 
tor.  These  factors  can  be  combined  to  yield  a  single  cQAF  for  adjustment  or 
maybe  used  independently  as  the  sole  QIP®  adjustment  factor  (Finkenstadt, 
2015).  This  process  may  become  agency-  and/or  service  industry-dependent, 
and  should  be  considered  by  agencies  prior  to  implementation. 

Once  the  agency  calculates  the  cQAF  for 
each  offeror,  the  agency  would  apply  the 
cQAF  to  the  total  price  of  each  line  item 
within  the  offeror’s  proposal  (Finkenstadt, 

2015).  The  agency  would  then  award  to 
the  conforming  offeror  demonstrating  the 
best-quality  offer  in  terms  of  both  price 
and  quality  ratings— in  other  words,  the 
lowest  evaluated  QIP®  offer. 

An  example  involving  advisory  services 
is  shown  in  Table  3.  In  this  example,  the 
cQAF  of  0.962  is  derived  from  the  calcu¬ 
lations  in  Table  1  by  rolling  up  a  notional 
cSVI  at  a  relative  importance  weighting 
of  30  percent  and  a  notional  cPQR  at  a 
relative  importance  weighting  of  70  per¬ 
cent.  The  30  percent  weight  on  cSVI  and  70  percent  weight  on  cPQR  are 
notional;  the  agency  would  determine  these  weights  depending  on  what 
is  more  important— actual  service  quality  from  past  work  or  promises  of 
future  service  quality  in  the  proposal.  The  cSVI  rating  of  0.976  is  created 
by  weighting  the  scores  of  each  primary  factor  of  reliability,  assurance, 
and  responsiveness  to  requirements.  The  cPQR  rating  of  0.956  is  derived 
by  weighting  the  firm’s  process  management  plan  and  staffing  quality. 
Again,  all  weightings  are  notional  and  would  be  established  prior  to  devel¬ 
oping  the  final  cQAF.  Using  this  example,  the  offer  would  be  assessed  as 
having  an  inherent  quality  value  of  $161,647.79.  Award  would  be  made  for 
$4, 273,570.00,  but  the  offeror  would  get  “credit”  for  having  a  lower  proposed 
price  based  on  carrying  a  higher  quality  rating  (cQAF).  This  is  the  final  step 
in  “monetizing”  the  trade-off  (Finkenstadt,  2015). 


Defense  ARJ,  April  2016,  Vol.  23  No.  2: 202-230 


215 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


0 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

O 

»- 

a, 

o 

o 

o 

o 

o 

o 

o 

o 

d 

o 

o 

d 

o 

o 

o 

o 

o 

o 

o 

o 

o 

tH- 

CD 

IS 

LO 

CD 

TJ 

o 

o 

o 

o 

o 

o 

' — „ 

CD 

Nj 

Mt 

in 

o 

0 

x 

o 

o 

o 

o 

CM 

CD~ 

IO 

hO 

00 

cd~ 

CO 

kT 

o 

hO 

o 

o 

CD 

CM 

(D 

[\ 

is 

5 

CD 

LL 

CD 

to 

CD 

CO 

hs 

co 

f\ 

m 

00 

IN 

CO 

q 

E 

to- 

co 

co 

CO 

CO 

CO 

hO 

hi 

W 

CO 

-1 

LL 

< 

z 

0 

*4 

o 

P 

H 

o 

0 

o 

O 

o 

O 

O 

o 

o 

O 

O 

o 

Z 

U 

o 

O 

o 

o 

o 

O 

o 

o 

LU 

T 

o 

O 

o 

o 

o 

o 

Mt 

CD 

00 

CL 

o 

O 

o 

o 

m- 

CM 

M- 

LO 

IO 

CO 

2 

■o 

0 

X 

o 

■tt 

to- 

o 

CM 

q 

to- 

^t 

CD~ 

CM 

CD 

'st 

CO 

o 

O) 

CM 

to 

LO 

CO 

q 

ro 

CO 

CO 

LO~ 

CO 

lo" 

hO 

CO 

LL 

CO 

CO 

CO 

z 

c 

o 

£ 

e 

h- 

LL 

< 

*4 

_l 

"E 

D 

O 

D 

_l 

< 

</) 

4-1 

u 

4—* 

4— > 

4— > 

4— 1 

4-» 

4-J 

4-J 

4-J 

4-» 

4-* 

© 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

/■n 

E 

_l 

_l 

_l 

_l 

_l 

_l 

_l 

_l 

_l 

_] 

a 

D 

O 

w 

> 

111 

*4 

y 

c 

LO 

LO 

LO 

LO 

LO 

LO 

LO 

LO 

LO 

LO 

a 

0 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

CM 

a. 

3 

a 

o 

LU 

(/> 

D 

LL 

Z 

> 

c 

c 

H 

o 

o 

-I 

< 

D 

4-J 

a 

O 

4-J 

a 

O 

o 

oS 

03 

a 

o 

CM 

O 

LU 

o 

O 

LL 

IL 

0 

a 

o 

O 

O 

/  \ 

/  s 

CM 

/~N 

CM 

IO 

/•“s 

ro 

/— \ 
Mt 

z 

_i 

z 

_l 

ro 

c 

u 

CD 

u 

(D 

LU 

c 

0 

✓”\ 

CD 

if) 

CO 

s~\ 

CD 

c 

o 

c 

o 

C 

o 

C 

o 

c 

o 

L_ 

o 

C 

o 

c 

o 

a 

M 

J 

CO 

Ul 

if) 

CO 

4— » 

a 

4—* 

Q. 

4-J 

Q. 

4-J 

a 

4-J 

Q. 

4-J 

a 

4— ' 

Q. 

4-J 

a 

w 

k 

U 

\_ 

u 

£ 

c 

o 

CO 

' — - 

CO 
' — ✓ 

O 

V—/ 

o 
\ _ ' 

O 
' — ' 

O 

V—/ 

O 
' — ' 

O 
' — - 

o 

v—/ 

O 

CL 

■o 

Q_ 

"D 

Q_ 

"D 

'& 

< 

< 

CO 

CO 

(J 

(J 

Q 

Q 

LU 

LU 

o 

I/I 

o 

a 

2 

(D 

0) 

0 

3 

< 

< 

CM 

< 

< 

CM 

< 

< 

CM 

< 

< 

CM 

< 

< 

CM 

O 

O 

U 

o 

O 

o 

O 

o 

O 

O 

O 

O 

O 

Q. 

Q. 

CD 

o 

O 

o 

O 

o 

O 

O 

O 

O 

O 

O 

O 

o 

o 

O 

o 

O 

o 

O 

O 

O 

o 

O 

a 

Q_ 

CL 

© 

a 

z 

z 

z 

z 

z 

z 

z 

z 

z 

z 

TO 

75 

75 

_l 

_l 

_l 

_] 

_l 

_l 

_l 

_l 

_J 

_] 

44 

o 

4-J 

o 

4-J 

o 

O 

u 

<J 

u 

u 

u 

(J 

u 

u 

u 

o 

£ 

F 

LO 

CM 

CD 

lo 

s 

M1 

K 

co 

LO 

CO 


O 

LO 

LO 

LO 

[\ 

O 

LO 

CM 

LO 

ro 

CO 


CM 

c\i 

CM 

CD 


Mf- 

co 


tD 

5 

J* 


o3 

O 

O 

O 


o3 

CM 

O 

O 

O 


o 

(s~s 

Q_ 

O 
_ _ • 

CD 

U 


~G 

CD 

c/> 

D 


•  (/) 
>  C 
~  O 

CD  4-J 

d  a 
O  O 


o 

(S~\ 

Q_ 

O 


"O 

CD 

if) 

=3 


i  (/) 
>  C 

~  o 

CD  4-J 

d  a 
O  O 


O 


216 


Defense ARJ,  April 2016,  Vol.  23 No.  2:202-230 


April  2016 


Case  Study 

A  case  study  methodology  was  used  to  test  a  portion  of  the  cQAF  in  a 
recent  source  selection  for  administrative  support  services.  This  task  order 
included  two  line  items  for  each  year  of  a  5-year  service  contract:  one  line 
item  for  contract  support  services  and  a  second,  larger  line  item  for  program 
control  (financial)  analysis  services.  The  case  involved  evaluating  offerors 
for  task  order  awards  under  a  prepriced  indefinite  delivery-indefinite  quan¬ 
tity  (IDIQ)  contract  in  which  11  offerors  could  offer  better  than  on-contract 
pricing,  but  had  to  offer  no  higher  than  on-contract  pricing.  The  request  for 
proposal  allowed  the  agency  to  decide  on  awarding  task  orders  for  one  or  all 
of  the  line  items.  The  IDIQs  did  not  allow  for  past  performance  evaluations 
in  the  base  year  of  the  IDIQs,  as  the  agency  considered  past  performance 
during  the  base  award  to  be  at  least  satisfactory  for  all  contractors  and 
prohibited  further  past  performance  evaluations  until  the  end  of  the  IDIQ 
base  period.  This  meant  that  all  trade-offs  for  nonprice  factors  could  not 
utilize  past  performance;  therefore,  the  cSVI  could  not  yet  be  tested.  The 
agency  chose  this  IDIQ  for  a  limited  test  case  due  to  (a)  perceived  weak¬ 
nesses  in  proposal  quality,  (b)  post-award  performance  results  on  recently 
LPTA-awarded  task  orders,  and  (c)  a  low  threat  to  mission  if  the  evaluation 
methodology  were  found  to  be  flawed  or  was  contested. 

Only  three  of  the  11  IDIQ  holders  submitted  a  proposal.  The  overall  assessed 
quality  of  these  three  proposals  was  relatively  high  compared  to  historical 
LPTA  evaluations  for  similar  services.  The  lowest  priced  offeror  was  not 
selected  due  to  having  the  lowest  cPQR  quality  rating.  Since  the  agency 
stated  that  quality  was  considered  more  important  than  price,  and  it  did 
not  intend  to  enter  discussions,  award  was  made  to  the  highest  rated  offeror 
in  terms  of  quality.  All  offerors  proposed  pricing  at  or  below  those  listed  in 
the  base  IDIQ.  The  lowest  offeror’s  pricing  was  perceived  as  questionably 
low  and  would  have  driven  the  team  into  discussions  had  their  quality 
rating  been  higher.  This  source  selection  did  not  fully  apply  the  concept  of 
adjusting  evaluated  pricing  since  it  was  a  first  trial.  The  researcher  first 
wanted  to  determine  whether  the  quality  rating  system  would  affect  the 
quality  trade-off. 

In  this  case,  the  highest  priced  yet  highest  rated  offeror  was  selected,  while 
the  overall  price  remained  4  percent  below  the  agency’s  estimate.  The  cPQR 
method  allowed  for  a  team  of  three  personnel  to  assess  three  full  proposals 
in  only  3  days.  Quality  perceived  was  converted  to  a  rating  that  yielded  the 
results  shown  in  Table  4.  Actual  cPQR  scores  are  not  available  due  to  source 
selection  material  and  the  sensitive  nature  of  the  procurement.  However, 
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this  table  demonstrates  their  relative  placement  after  applying  the  cPQR 
as  a  general  quality  ranking  independent  of  QIP®  adjustment.  The  full  QIP® 
methodology,  applying  the  cQAF  to  price  in  order  to  rank  offerors  based 
on  QIP®,  was  not  completed  in  the  actual  source  selection.  The  agency 
surmised  that  limited  application  of  the  methodology  would  minimize 
industry  confusion  over  an  unfamiliar  evaluation  methodology,  but  would 
allow  for  early  testing  of  the  concept.  This  case  was  a  first-off  trial,  and 
this  research  calls  for  agencies  to  consider  future  and  full  application  of 
the  QIP®  methodology. 


j  TABLE  4.  CASE  STUDY  TRADE- 

■OFF  EVALUATION  l 

Rating  Factor 

Offeror  X 

Offeror  Y 

Offeror  Z* 

Price  (CLIN  0001) 

1  of  3 

3  of  3 

2  of  3 

Price  Variance  from  Low 

0% 

37% 

28% 

Price  (CLIN  0002) 

1  of  3 

2  of  3 

3  of  3 

Price  Variance  from  Low 

0% 

14% 

25% 

Nonprice  (cPQR) 

CLIN  0001 

3  of  3 

2  of  3 

1  of  3 

Nonprice  (cPQR) 

CLIN  0002 

2  of  3 

3  of  3 

1  of  3 

•Awarded  offeror 

Results 

Although  no  evaluated  price  calculations  were  made  during  the  actual 
source  selection  using  QIP®,  the  scaling  methodology  was  maintained  to 
allow  the  researcher  to  take  actual  source  selection  data  and  run  the  scenario 
utilizing  the  full  QIP®  scoring  concept  to  identify  strengths  and  weaknesses 
in  the  concept  post  hoc  so  as  not  to  affect  the  actual  award  decision.  In  this 
application,  the  researcher  applied  a  cPQR  created  by  using  the  program 
management  plan  and  the  staffing  quality  criteria  similar  to  Table  1,  yet 
heavily  customized  for  each  type  of  service  being  procured.  The  weightings 
of  the  plan  versus  staffing  quality  are  source  selection-sensitive,  but  offerors 
were  told  which  held  the  greatest  importance  to  aid  in  proposal  development. 

Quality  trade-offs  in  a  trial  service  contract  source  selection  resulted  in 
useful  measures  of  service  quality,  cooperative  industry  participation,  fewer 
and  higher  quality  proposals  (i.e.,  less  risk  of  adverse  selection  and  greater 
efficiency),  and  a  different  contractor  selection  than  a  typical  subjective 
price-performance  trade-off.  The  effectiveness  of  this  QIP®  methodology 
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must  be  supported  by  scaling  that  considers  relative  price  variations  in  the 
market.  Had  QIP®  been  applied  to  the  actual  source  selection,  the  lowest 
quality  offeror  having  the  lowest  evaluated  price  may  have  been  selected 
because  the  quality  scaling  was  not  sufficient  to  overcome  the  wide  variation 
in  pricing.  Discussions  would  have  been  necessary  to  determine  the  rea¬ 
sonableness  of  the  lowest  offeror’s  pricing  to  reduce  the  risk  of  “buying-in.” 
These  discussions  would  have  been  completed  prior  to  applying  the  QIP® 
to  ensure  that  the  final  results  were  accurate. 

When  applying  QIP®,  the  test  case  shows  that  the  selected  offeror  becomes 
more  competitive  in  terms  of  evaluated  price,  relative  to  the  low,  based  on 
evaluated  quality,  and  does  in  fact,  displace  the  second  lowest  offeror  from 
the  non-QIP®  evaluation  (Table  5).  Thus,  the  QIP®  methodology  demon¬ 
strates  the  ability  to  drive  value  for  money  into  an  evaluation  and  to  create 
source  selection  results  that  more  closely  align  to  traditional,  yet  more  sub¬ 
jective,  full  trade-off  methods  in  a  rapid  manner  that  is  more  transparent 
and  easier  to  use. 


'  TABLE  5.  CASE  STUDY  QIP 

®-ADJUSTED  TRADE-OFF  EVALUATION  | 

Rating  Factor  (After  QIP®) 

Offeror  X** 

Offeror  Y 

Offeror  Z* 

QIP®  (CLIN  0001) 

1  of  3 

3  of  3 

2  of  3 

Price  Variance  from  Low 

0% 

33%  * 

16%  * 

QIP®  (CLIN  0002) 

1  of  3 

3  of  3  * 

2  of  3  * 

Price  Variance  from  Low 

0% 

17%  * 

12%  * 

‘Awarded  offeror 

”QIP®  best-value  offeror 

To  validate  results,  a  questionnaire  was  sent  to  all  quality  team  evaluators 
postaward  (Appendix  B).  The  responses  indicated  that  all  evaluators  found 
the  cPQR  methodology  easy  to  use,  easy  to  understand,  asked  the  right  types 
of  questions,  and  resulted  in  the  best  value  to  the  government.  The  only  area 
listed  for  cPQR  improvement  related  to  requiring  the  evaluation  team  to 
have  earlier  and  more  robust  input  into  the  relative  importance  weighting 
of  cPQR  categories.  The  ease  of  use  and  ability  for  lesser  trained  personnel 
to  administer  this  methodology  show  significant  promise  in  reducing  the 
risk  to  poorly  executed,  best-value  trade-off  evaluations  that  can  occur  due 
to  less  experienced  evaluators  (Watson,  2015). 
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Discussion 

The  Figure  demonstrates  that  while  LPTA  provides  for  a  faster  PALT 
and  is  relatively  transparent,  it  sacrifices  value  for  money  in  service  acqui¬ 
sitions.  While  full  trade-off  has  the  capability  to  maximize  value  for  money, 
it  may  reduce  transparency  if  evaluations  become  too  complex,  and  most 
assuredly  will  sacrifice  speed  of  the  service  acquisition.  QIP®,  as  proposed, 
would  provide  a  means  for  monetizing  trade-offs.  Monetizing  trade-offs 
prevents  pre-award  questions  related  to  full  trade-off  ambiguity  as  well  as 
post-award  delays  due  to  protest.  The  QIP®  provides  for  faster  acquisition 
of  needs,  with  a  clearer  evaluation  methodology  and  trade-offs  that  increase 
both  value  for  money  and  transparency. 


FIGURE.  COMPARISON  OF  TRADE-OFF  METHODS  RELATIVE  TO 
PRIMARY  PUBLIC  PROCUREMENT  OBJECTIVES 
WHEN  PROCURING  SERVICES 


Best  Value  Continuum  with  QIP®  (Notional) 


LPTA  QIP1  Full  TO 


The  use  of  QIP®  has  the  potential  to  improve  or  eliminate  major  gaps  found 
in  the  current  best -value  source  selection  process  (Watson,  2015)  for  ser¬ 
vices.  QIP®  provides  the  following: 

•  Monetized  trade-offs  (Transparency) 

•  Ability  to  pay  more  for  service  quality  when  prudent  (Value 
for  Money) 

•  Clear  communication  of  federal  agency  priorities  in  price  and 
nonprice  factors  to  offerors  (Transparency) 
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•  May  help  to  correct  for  wide  price  disparities  in  previously 
negotiated  multiple-award  contracts  (i.e.,  when  the  low  would 
otherwise  always  win  if  LPTA  were  the  only  option  to  full 
trade-off)  (Value  for  Money) 

•  Rapid  evaluation  and  acquisition  capability  (Meeting 
Requirements/Need  with  Speed) 

•  Clear  evaluation  criteria  that  reduce  protest  risk  (Transparency) 

In  addition,  the  QIP®  cSVI  component,  as  a  crowd-sourced  form  of  past 
performance,  has  several  advantages: 

•  Encourages  higher  compliance  rates  for  past  performance 
reporting  by  providing  a  clear,  easy-to-use  format  with  more 
resemblance  of  commercial,  crowd-sourced  contractor  per¬ 
formance  reporting 

•  Fills  past-performance  assessment  repository  gaps 

•  Promotes  rapid  evaluation  and  acquisition  capability  (could 
replace  the  entire  past  performance  volume  requirement  in 
proposals) 

•  Encourages  better  life-cycle  performance  with  contractors 
(i.e.,  contractors  with  lower  cSVIs  will  have  price  advantages 
and  can  offer  higher  quality  services  assuming  a  better  QIP®, 
while  higher  cSVI  contractors  will  have  to  be  more  aggressive 
in  pricing  in  the  near-term  and  improve  quality  in  the  long¬ 
term  to  keep  market  share  and  realize  higher  future  returns) 

•  Encourages  pricing  off-sets  for  performance  issues 

•  Creates  clear  discriminators  for  services  based  on  customer 
ratings  (subjective  customer  quality  is  a  truer  way  to  assess 
the  intangibility  inherent  in  service  performance) 

•  Arms  federal  agencies  with  real-time  market  performance  data 

•  Enables  more  accurate  and  more  efficient  supplier  ranking 
(e.g.,  DoD  superior  supplier  incentive  program).  More  efficient 
rankings  will  enable  rankings  by  type  of  service  rather  than  be 
limited  to  the  top  30  business  units  by  dollars  obligated  annually 
(DoD,  2015). 
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Conclusions 

A  QIP®  methodology  using  an  established  cSVI  system  shows  great 
promise  in  progressing  the  state  of  the  art  in  contractor  performance  man¬ 
agement  while  finding  a  desirable  midpoint  along  the  best -value  continuum. 
This  research  calls  for  federal  agencies  to  consider  adopting  such  a  meth¬ 
odology  to  meet  public  procurement  objectives.  The  QIP®  and  cSVI  concepts 
may  be  seen  as  “lofty”  or  even  naive  from  a  historical  federal  procurement 
policy  vantage  point.  However,  that  vantage  point  is  built  upon  a  history  of 
ill-fated  service  contractor  rating  systems  that  never  meet  the  intent  of 
federal  agencies  to  improve  transparency,  value  for  money,  and  require¬ 
ments  satisfaction  in  highly  efficient  and  effective  means.  The  current 
CPARS  has  more  focus  on  gettingthe  reports  completed  versus  the  accuracy 
and  value  of  the  reports— particularly  for  its  intended  purpose  of  better 
informing  future  source  selections.  Understanding  the  higher  level  impacts 
of  the  system  as  it  relates  to  transparency,  value  for  money,  and  meeting 
agency  requirements  should  be  the  ultimate  goal  of  any  contractor  perfor¬ 
mance  rating  system  as  well  as  the  source  selection  process  it  feeds. 


Understanding  the  higher  level  impacts  of  the  system 
as  it  relates  to  transparency,  value  for  money,  and 
meeting  agency  requirements  should  be  the  ultimate 
goal  of  any  contractor  performance  rating  system  as 
well  as  the  source  selection  process  it  feeds. 

Critics  may  question  the  ability  to  adjust  an  offeror’s  evaluated  price  based 
on  subjective  evaluation  inputs.  However,  the  government  does  this  today 
with  the  concept  of  most  probable  price  and  cost  evaluations  to  determine 
what  the  agency  anticipates  the  actual  cost  or  price  of  an  offer  will  be  con¬ 
sidering  all  risks.  Considering  that  service  quality  is  a  primary  risk  concern 
in  a  services  acquisition,  the  concept  of  QIP®  is  not  a  radical  idea.  Others 
question  the  idea  of  crowd-sourcing  something  as  sensitive  as  contractor 
performance  for  federal  contract  award  decisions.  This  can  be  mitigated  by 
controlling  the  “crowd”  as  we  do  today  with  CPARS.  Agencies  should  ensure 
that  only  contracting  officers,  contracting  officer  technical  representatives, 
and  possibly  program  managers  have  access  to  the  cSVI  rating  system. 
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Limitations  and  Future 
Research  Directions 

This  study  is  not  without  limitations.  First,  it  is  a  limited  application  of 
one  case.  Future  research  could  expand  the  number  and  variety  of  cases  of 
application.  Future  research  employing  a  quasi-experiment  could  compare 
sourcing  and  performance  (i.e.,  the  full  service  life  cycle)  of  multiple  service 
procurements  of  the  same  type  of  service  to  examine  differences  in  value 
and  service  quality.  Further  research  should  also  explore  the  customization 
of  dimensions  of  the  business-to-business  service  quality  measurement 
scale.  Different  types  of  services  will  likely  be  more  validly  measured  by 
customized  aspects  of  service  quality.  Additionally,  since  different  ser¬ 
vices  span  a  vast  spectrum  of  scope  and  complexity,  further  research  could 
explore  whether  the  proposed  QIP®  methodology  will  be  equally  effective 
across  the  different  types  of  services.  In  closing,  this  article  serves  as  a  call 
to  agencies  to  pilot-test  the  QIP®  concept. 
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Appendix  A 

Comparison  of  Best-Value  Source  Selection  Options  Relative  to 
Primary  Public  Procurement  Objectives 


j  Objective 

LPTA 

Full  Trade-off 

Rating  Rationale  f 

Transparency 

High 

Low 

LPTA  typically  defines 
evaluation  criteria  in  a  very 
clear  and  objective  fashion. 
Protest  risk  is  minimized  if 

LPTA  process  is  followed. 

• 

Full  trade-off  may  allow  for 
high  levels  of  subjectivity  at 
the  factor  and  subfactor  level, 
and  runs  the  risk  of  being 
challenged  both  pre-  and 
post-award. 

Value  for 
Money 

Low 

High 

LPTA  clearly  states  the 
agencies’  desire  to  pay  less 
for  a  base  requirement  and 
no  more.  May  drive  “bare 
minimum”  solutions  from 
industry  in  an  effort  to  remain 
competitive.  High  risk  of 
“buying-in.” 

• 

Full  trade-off  establishes 
areas  of  trade  that  are 
primarily  quality-  and 
performance-based;  reduces 
the  risk  of  post-award 
performance  issues  (GAO, 
2014a). 

Meeting 
Agency 
Requirements 
(Need  with 

Low 

Medium 

LPTA  tends  to  meet  timelines 
and  basic  requirements.  Can 
be  risky  if  unknowns  surface 
postsolicitation. 

Speed) 

• 

Full  trade-off  expends  the 
greatest  amount  of  time  in 
order  to  minimize  the  risk 
of  unknowns  insofar  as  the 
evaluation  criteria  plans  for 
it  (i.e.,  assessing  proposal 
risk).  However,  minimizing 
unknowns  equates  to  unclear 
subjectivity  that  may  increase 
protest  risks. 

Note.  High  =  highest  level  relative  to  alternative;  Med 
alternative;  Low  =  lowest  level  relative  to  alternative. 

=  essentially  the  same  as  the 
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Appendix  B 

Post-award  cPQR  Evaluator  Questionnaire 

1.  How  easy/hard  did  you  find  the  criteria  to  understand? 

2.  Were  the  evaluation  tools  easy  to  use  or  hard  to  use? 

3.  Did  we  [the  agency]  ask  the  right  questions  in  the  evaluation  or  could 
we  have  done  better? 

4.  Is  there  anything  you  would  do  to  improve  this  evaluation  method  in 
the  future? 


Appendix  C 

Post-award  cPQR  Vendor  Questionnaire 

1.  Was  your  decision  to  propose  or  “no  bid”  [actual  term  used  within  the 
ordering  procedures  of  the  base  IDIQ]  based  on  the  cPQR  methodology 
used? 

2.  What,  if  anything,  did  you  change  about  your  traditional  proposal 
methods  in  order  to  meet  the  requirements  of  this  request  for  proposal’s 
cPQR  methodology? 

3.  Was  the  cPQR  evaluation  methodology  easy  to  understand? 

4.  What,  if  anything,  would  you  change  about  the  cPQR  evaluation  meth¬ 
odology  used? 

5.  Considering  your  experience,  in  the  future  would  you  be  open  to  having 
your  evaluated  price’1'  adjusted  based  on  the  score  received  using  a  sim¬ 
ilar  evaluation  methodology? 

6.  Did  the  cPQR  methodology  encourage  your  firm  to  focus  more  on  price 
or  nonprice  (i.e.,  quality)  factors  in  proposing? 


*Note.  The  price  would  be  adjusted  for  evaluation  purposes  only.  The  final  award  price 
would  be  as  proposed  or  negotiated  with  the  [agency]. 
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Appendix  D 

Findings  and  Lessons  Learned 


Finding  Lessons  Learned 


Finding 

Lessons  Learned 

The  cPQR  primary  factors  and 
subfactor  areas  and  questions 
were  highly  indicative  of  a  quality 
proposal  from  the  requiring 
agencies’  point  of  view 

Procuring  agencies  should  develop 
cPQR  satisfaction  questions  in  tight 
coordination  with  the  requiring 
activity.  Standardized,  valid  cPQR 
measures  customized  to  each  type 
of  service  could  evolve  over  time. 

The  scaling  factor  used  ranged 
from  0.85  to  1.15,  but  was  not 
indicative  of  the  potential  pricing 
variations  across  the  service  line- 
item  disciplines  being  proposed, 
and  was  based  more  on  what  the 
procuring  agency  calculated  were 
rational  price  variations  in  typical 
procurements. 

Procuring  agencies  should  develop 
the  scaling  factor  ranges  based 
on  market  research  into  the 
commercial  market’s  typical  price 
variation  across  each  service  type, 
and  not  assume  what  a  “fair”  scaling 
should  be.  Note:  This  confirmed 
the  risk  to  utilizing  QIP®  price 
adjustments  when  the  scaling  has 
not  been  developed  based  on 
robust  market  data. 

Industry  did  not  question  the  unique 
quality  evaluation  methodology. 

Draft  request  for  proposal  (RFP) 
documents  were  posted  to  the 
business  opportunity  Web  site  to 
gather  questions  from  industry 
and  ensure  it  understood  the 
methodology  prior  to  issuing 
a  final  RFP.  Draft  RFPs  and 
industry  engagement  are  key 
when  introducing  new  evaluation 
methods. 

Only  roughly  a  third  of  contractors 
on  the  Multiple  Award  IDIQ 
proposed  to  provide  these  services 
to  the  agency.  Most  no-bid  letters 
received  by  the  agency  stated  an 
inability  to  source  personnel  who 
met  the  quality  requirements  of 
the  RFP.  Anecdotal  comparisons  of 
number  of  offers  received  on  similar 
RFPs  under  this  IDIQ  showed  that 
LPTA  yielded  higher  response  rates 
with  lower  quality  offers. 

The  agency  received  offers  from 
only  those  contractors  who  could 
meet  the  agencies’  desired  quality 
needs.  The  natural  desire  found  in 
LPTA  to  “buy-in”  was  minimized. 

The  risk  of  adverse  selection  was 
mitigated  and  the  selection  was 
more  efficient. 

228 


Defense ARJ,  April 2016,  Vol.  23 No.  2:202-230 


April  2016 


Appendix  D,  Continued 


comparison  of  cPQR  scores  versus 
prices  offered. 


The  agency  would  have  reached  a 
different  award  decision  had  the 
QIP®  been  utilized  versus  leaving 
the  final  trade-off  to  a  subjective 


Agencies  must  ensure  they 
have  robust  data  to  support  the 
cQAF  primary  categories  and 
subcategory  questions,  and  a  solid 
understanding  of  the  scaling  of 
each  rated  area  and  the  associated 


weights  to  create  meaningful 
discrimination  between  offers  within 
a  service  type.  Further,  the  use  of 
cSVI  would  have  had  additional 
effects  on  the  final  scoring  and 
should  be  considered  in  all  future 
QIP®  source  selections  where  past 
performance  is  being  evaluated. 
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Biographies 


Capt  Daniel  J.  Finkenstadt,  USAF,  is  a  pro¬ 
curing  contracting  officer  serving  at  the  National 
Reconnaissance  Office  (NRO)  in  Chantilly, 
Virginia.  He  has  12  years’  contracting  experience 
and  is  responsible  for  one  of  NRO’s  largest  major 
systems  acquisitions.  Capt  Finkenstadt  has 
authored  six  publications  related  to  contract  man¬ 
agement.  He  holds  a  BS  in  Business  Management 
from  Mt.  Olive  College  and  an  MBA  in  Strategic 
Sourcing  from  the  Naval  Postgraduate  School. 


(E-mail  address:  asuscrash@gmail.com) 


Lt  Col  Timothy  G.  Hawkins,  USAF  (Ret.), 

is  an  assistant  professor  at  Western  Kentucky 
University.  He  has  published  18  articles  exploring 
procurement  and  logistics  in  scholarly  journals. 
Lt  Col  Hawkins  holds  a  PhD  in  Marketing  from 
the  University  of  North  Texas,  an  MS  from 
Wilmington  College,  and  aBS  in  Civil  Engineering 
Technology  from  Western  Kentucky  University. 
While  on  active  duty,  he  served  as  a  warranted 
contracting  officer  for  the  U.S.  Air  Force. 


(E-mail  address:  timothy.hawkins@wku.edu) 
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suggested  books,  with  more  available 
on  our  Web  site  http://dau.dodlive. 
mil/defense-acquisition-profession- 
al-reading-program. 


We  encourage  our  readers  to  submit 
book  reviews  they  believe  should 
be  required  reading  for  the  defense 
acquisition  professional.  The  books 
themselves  should  be  in  print  or 
generally  available  to  a  wide  audi¬ 
ence;  address  subjects  and  themes 
that  have  broad  applicability  to 
defense  acquisition  profession¬ 
als;  and  provide  context  for  the 
reader,  not  prescriptive  practices. 
Book  reviews  should  be  450  words 
or  fewer,  describe  the  book  and  its 
major  ideas,  and  explain  its  rele¬ 
vancy  to  defense  acquisition.  Please 
send  your  reviews  to  the  managing 
editor,  Defense  Acquisition  Review 
Journal  at  DefenseARJ@dau.mil. 


Featured  Book 


The  American  Warfare  State:  The  Domestic 
Politics  of  Military  Spending 
Author:  Rebecca  U.  Thorpe 
Publisher:  University  of  Chicago  Press 

Copyright  Date:  2014 
Hard/Softcover/Digital:  Available  in  all 
three  media,  245  pages 

ISBN: 

ISBN:  9780226124070  (Softcover) 

ISBN:  9780226123912  (Hardcover) 

ISBN:  9780226124100  (e-Book)  Available 
online  at  http://press.uchicago.edu/ucp/ 
books/book/chicago/A/bol7607479.html 
Reviewed  by:  Professor  Trevor  Taylor, 
Cranfield  University,  Defence  Academy  of  the 
United  Kingdom 


April  2016 


Review: 

Most  observers  would  recognize  that  the  size  and  shape  of  the  U.S. 
defense  budget  is  determined  by  more  factors  than  the  perceived  threats 
to  U.S.  security  and  the  endorsed  strategy  to  manage  them.  This  import¬ 
ant  book  focuses  largely  on  one  consideration:  the  concern  of  members  of 
Congress  with  defense-related  employment  in  their  constituencies. 

The  broad  propositions  of  the  work  are  that  during  World  War  II,  defense 
production  became  more  central  to  the  U.S.  economy  and  moved  into  new 
regions.  Not  least  among  these  were  rural  areas  with  few  economic  opportu¬ 
nities.  After  1945,  there  was  a  failure  to  cut  defense  spending  on  a  significant 
scale,  and  more  areas  became  structurally  dependent  on  defense-related 
jobs.  With  defense  contractors  increasingly  aware  that  congressional 
representatives  support  programs  that  bring  jobs  to  their  districts,  sub¬ 
contracting  on  major  programs  has  been  spread  farther  across  the  country. 
The  resultant  wider  constitutional  consequences  are  that  Congress  has 
largely  given  up  its  constitutional  role  as  a  second  center  of  decision  mak¬ 
ing  regarding  the  size  of  the  defense  budget  and  whether  the  United  States 
should  commit  to  the  use  of  military  force. 

The  author  has  researched  diligently  in  search  of  statistical  correlations 
to  support  her  arguments,  particularly  regarding  voting  patterns  and  the 
geographical  dispersion  of  subcontracts.  She  has  assembled  a  significant 
evidence  base  showing  that  “the  shared  threat  of  economic  hardship  affects 
legislative  voting  on  targeted  and  generalized  weapons  spending”  (p.  106). 

Although  the  following  points  are  not  made  by  the  author,  U.  S.  defense  con¬ 
tracting  appears  to  have  some  parallels  with  the  “juste  retour”  principle, 
whereby  the  proportion  of  contracts  under  a  particular  program  awarded  to 
firms  from  a  given  country  is  in  proportion  to  the  funding  that  country  has 
contributed  to  the  program.  This  principle  tends  to  operate  on  collaborative 
weapons  programs  in  Europe  and  even  with  offset  demands  made  by  many 
arms  purchasing  states.  Legislators  in  many  states  prefer  to  see  the  pain 
of  defense  procurement  spending  reduced  by  ensuring  such  expenditures 
generate  as  much  local  economic  benefit  as  possible. 

The  work  is  not  without  flaws:  in  particular,  the  ongoing  sequestration  expe¬ 
rience  is  not  analyzed  and  the  author  does  not  venture  into  big  questions 
that  the  book’s  core  arguments  will  suggest  to  some  readers.  Is  the  division 
of  powers  advocated  by  Founding  Father  James  Madison  appropriate  in 
the  modern  age  when  speed  of  decision  or  credibility  of  commitment  may 
be  of  greater  importance?  How  serious  was  the  Soviet/Communist  threat 
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after  1945  that  gave  public  justification  for  the  continued  defense  effort? 
Is  it  an  inevitable  feature  of  the  capitalist  system  in  the  United  States  that 
some  rural  areas  will  remain  underdeveloped  compared  with  other  areas 
of  the  country?  Linked  to  all  this  is  that  the  book  is  short  on  prescriptions 
for  improving  the  situation,  and  there  is  no  discussion  of  the  defense  indus¬ 
trial  “conversion”  efforts  that  occurred  in  Europe  and  elsewhere  after  1990. 
Instead,  the  book  is  focused  on  building,  in  a  terse  style,  a  few  significant 
arguments  and  effectively  reinforces  the  broad  point  that  defense  acquisi¬ 
tion  is  as  much  about  politics  as  it  is  about  management  techniques. 
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CALL  FOR  AUTHORS 


We  are  currently  soliciting  articles  and  subject  matter 
experts  for  the  2016  Defense  Acquisition  Research  Jour¬ 
nal  ( ARJ)  print  year.  Please  see  our  guidelines  for  con¬ 
tributors  for  submission  deadlines. 


Even  if  your  agency  does  not  require  you  to  publish,  consider  these  career-enhancing 
possibilities: 

•  Share  your  acquisition  research  results  with  the  Acquisition,  Technology,  and  Logistics 
(AT&L)  community. 

•  Change  the  way  Department  of  Defense  (DoD)  does  business. 
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program. 
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implementation  of  new  initiatives. 

•  Condense  your  graduate  project  into  something  beneficial  to  acquisition  professionals. 
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•  Earn  25  continuous  learning  points  for 
publishing  in  a  refereed  journal. 

•  Earn  a  promotion  or  an  award. 

•  Become  part  of  a  focus  group  sharing 
similar  interests. 

•  Become  a  nationally  recognized  expert 
in  your  field  or  specialty. 

•  Be  asked  to  speak  at  a  conference 
or  symposium. 


We  welcome  submissions  from  anyone  in¬ 
volved  with  or  interested  in  the  defense  ac¬ 
quisition  process— the  conceptualization, 
initiation,  design,  testing,  contracting,  pro¬ 
duction,  deployment,  logistics  support,  mod¬ 
ification,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  (including  con¬ 
struction)  needed  by  the  DoD,  or  intended  for 
use  to  support  military  missions. 


If  you  are  interested,  contact  the  Defense  ARJ  managing  editor  (DefenseARJ@dau.mil)  and 
provide  contact  information  and  a  brief  description  of  your  article.  Please  visit  the  Defense  ARJ 
Guidelines  for  Contributors  at  http://www.dau.mil/pubscats/Pages/ARJ.aspx, 
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The  Defense  ARJ  is  published  in  quarterly  theme  editions.  All  submis¬ 
sions  are  due  by  the  first  day  of  the  month.  See  print  schedule  below. 
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In  most  cases,  the  author  will  be  notified  that  the  submission  has  been 
received  within  48  hours  of  its  arrival.  Following  an  initial  review,  submis¬ 
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